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Abstract 

For any Hcrmitian Lie group G of tube type we give a geometric 
quantization procedure of certain Xc-orbits in pj to obtain all scalar 
type highest weight representations. Here Kc is the conrplcxihcation 
of a maximal compact subgroup K C G with corresponding Cartan 
decomposition q = f + p of the Lie algebra of G. We explicitly real- 
ize every such representation 1 on a Fock space consisting of square 
integrable holomorphic functions on its associated variety Ass(7r) C pj. 

The associated variety Ass(7r) is the closure of a single nilpotent 
K<c-orb\t O c C pp which corresponds by the Kostant-Sekiguchi cor- 
respondence to a nilpotent coadjoint G-orbit O g C q* . The known 
Schrddinger model of 7r is a realization on L 2 (0), where O C O g 
is a Lagrangian submanifold. We construct an intertwining operator 
from the Schrddinger model to the new Fock model, the generalized 
Segal-Bargmann transform, which gives a geometric quantization of 
the Kostant-Sekiguchi correspondence (a notion invented by Hilgcrt, 
Kobayashi, 0rsted and the author). 

The main tool in our construction are multivariable 7- and K- 
Bessel functions on Jordan algebras which appear in the measure of 
1 c , as reproducing kernel of the Fock space and as integral kernel 
of the Segal-Bargmann transform. As a corollary to our construction 
we also obtain the integral kernel of the unitary inversion operator in 
the Schrddinger model in terms of a multivariable J-Bcssel function as 
well as explicit Whittakcr vectors. 
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Introduction 



The unitary highest weight representations of a semisimple Lie group G were 
classified by Enright-Howe-Wallach [13] and Jakobsen [29] independently 
(see also [14]). Among them are the scalar type representations which were 
first completely described by Berezin [6] for the classical groups and by 
Wallach [50j in the general case and are parameterized by the so-called 
Wallach set (sometimes also referred to as the Berezin- Wallach set). There 
are various different realizations of these representations, e.g. on spaces of 
holomorphic functions on a bounded symmetric domain. Rossi- Vergne |49j 
gave a realization which can by the work of Hilgert-Neeb-0rsted [25} [26l [27] 
be understood as the geometric quantization of a nilpotent coadjoint G-orbit. 
More precisely, their model lives on the space L 2 (0) of square integrable 
functions on a Lagrangian submanifold O C O g of a nilpotent coadjoint 
G-orbit O g C g* in the dual of the Lie algebra g of G. We refer to this 
realization as the Schrodinger model. 

The Kostant-Sekiguchi correspondence asserts a bijection between the 
set of nilpotent coadjoint G-orbits in g* and the set of nilpotent i^c-orbits in 
p%, , where Kq is the complexification of a maximal compact subgroup K C G 
of G with corresponding Cartan decomposition g = t + p. Under this bijec- 
tion O g corresponds to a nilpotent i^c-° r bit Kc in p£. Harris-Jakobsen 
[19j obtained a realization of each scalar type unitary highest weight repre- 
sentation of g on regular functions on the corresponding associated variety 
in p£ (see also [30]). However, their construction is not explicit and of purely 
algebraic nature. They neither give a geometric construction of the invari- 
ant inner product nor do they analytically describe the representation space 
for the group action. 

However, for the special case of the even part of the metaplectic represen- 
tation, which is a highest weight representation of scalar type of g = sp(k, K), 
a geometric quantization of the corresponding /fc-orbit is well-known. It 
gives the even part J 7 even (C k ) of the classical Fock space, consisting of even 
holomorphic functions on C k which are square integrable with respect to 
the Gaussian measure e - ' 2 ' dz. In this case also the intertwining oper- 
ator between the Schrodinger model on L 2 vcn (M. k ) , the space of even L 2 - 
functions on W k , and the Fock model J- even (C k ) is explicit. It is given by 
the classical Segal-Bargmann transform which is the unitary isomorphism 
E : L 2 vcn {R k ) ^ J- cvcn (C fc ) given by 

Bij(z) = e-^ z2 [ e 2z - x e~ x \(x)dx. (0.1) 

For Hermitian groups of tube type, we construct in a completely analytic 
and geometric way analogues of the Fock space and the Segal-Bargmann 
transform for any unitary highest weight representation of scalar type. The 
generalized Fock space consists of holomorphic functions on Q Kc which are 
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square integrable with respect to a certain measure. This establishes a 
geometric quantization of the nilpotent i£c-° r bit Kc . We further find 
explicitly the intertwining operator between this new Fock model and the 
Schrodinger model, the generalized Segal-Bargmann transform. This inte- 
gral transform can be understood as a geometric quantization of the Kostant- 
Sekiguchi correspondence, a notion invented in [23]. We remark that the 
special case of the minimal nilpotent .fTc-orbit was treated in [23] which was 
in fact the starting point of our investigations. 

Although some of our proofs are obvious generalizations of the proofs 
in [23] for the rank 1 case (the minimal scalar type highest weight repre- 
sentation), the general case requires new techniques. Whereas in [23] it 
was possible to work with classical one-variable Bessel functions, we have 
to use multivariable J-, /- and X-Bessel functions on Jordan algebras in 
the general case. These were studied before in [8] [10], [11] [12] [16], partly 
in a different context. We systematically investigate them further, proving 
asymptotic expansions, growth estimates, invariance properties, differential 
equations, integral formulas and their restrictions to the -K"c-° r bits Kc . We 
then use the if-Bessel functions in the construction of the measures on the 
orbits Kc and the /-Bessel functions as reproducing kernels of the Fock 
spaces and as integral kernels of the Segal-Bargmann transforms. Addi- 
tional results not studied in [23] involve a detailed analysis of corresponding 
-ftTc-orbits and rings of differential operators on them (Section I4.5p . the in- 
tertwining operator to the bounded symmetric domain model (Section 15. 2p . 
Whittaker vectors in the Schrodinger model (Section I6.2p and applications 
to branching laws (Section [7]). We further believe that our more general 
construction extends the known realizations of unitary highest weight rep- 
resentations in a natural and organic way. 

We now explain our results in more detail. Let G be a simple connected 
Hermitian Lie group of tube type with trivial center. Then G occurs as the 
identity component of the conformal group of a simple Euclidean Jordan 
algebra V. Its Lie algebra q acts on V by vector fields up to order 2. More 
precisely, we have a decomposition 

= n + [ + n, (0.2) 

where n = V acts on V by translations, [ = str(V) is the structure algebra 
of V acting by linear transformations and n acts by quadratic vectorfields. 
There is a natural Cartan involution i? on G given in terms of the inversion in 
the Jordan algebra. Denote by K = G^ the corresponding maximal compact 
subgroup of G and by g = t + p the associated Cartan decomposition. Then 
t has a one-dimensional center Z(t) = MZq. Let Kc be a complexification 
of K, acting on pc- Then pc decomposes into two irreducible .ffc- m odules 
p + and p~ and Zq G Z(t) can be chosen such that p^ 1 is the ±z eigenspace 
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of ad(Zo) on pc- We have the well-known decomposition 

gc = P"+*c + P + - (0.3) 

There is a Cayley type transform G € Int(gc) ( see (|l-6p for the precise 
definition) which interchanges the decompositions (j0.2j) and (j0.3f) in the 
sense that 

C(p~)=n c , C(*c) = fc, G(p+) = n c . 

Unitary highest weight representations 

Let (tTjT-L) be an irreducible unitary representation of the universal cover G 
of G and denote by (d7r,?^) its underlying (g, if)-module, K C G being 
the universal cover of X. We say that (tt,T-L) or ( d7r, "H^) is a highest weight 
representation if 

ft£ := {»£%: d7r(p+)v = 0} / 0. 

The space H~ is an irreducible ^-representation. If further dim%~ = 1 
we say that (7r, H) is of scalar type. In this case only the center Z($) = 
IRZo of I can act nontrivially on Ti- and the scalar type highest weight 

representations of G are parameterized by this scalar. More precisely, we 
define the Wallach set 

W = {o,|...,(r-l)^}u((r-l)^oo), 

where d is a certain root multiplicity of g and r denotes the rank of the 
Hermitian symmetric space G/K. Then for each A € W there is exactly one 
scalar type highest weight representation T~L of G in which Zq acts on T-L p ~ 
by the scalar — i^. 

Nilpotent orbits 

We identify g* with g and p c with pc via the Killing form of 0c- By the 
Kostant-Sekiguchi correspondence the nilpotent adjoint G-orbits in g are in 
bijection with the nilpotent i^c-orbits in pc- For a certain class of orbits we 
make this correspondence more precise. The analytic subgroup L of G with 
Lie algebra [ acts on V = n via the adjoint representation. It has finitely 
many orbits, among them an open orbit O C V (the orbit of the identity 
element in the Jordan algebra) which is a symmetric cone. Its closure has 
a stratification into lower dimensional orbits Co, • • • , O r -i with Ok Q Ofc+i- 
Put O r := f2. These orbits in n generate nilpotent adjoint G-orbits 

Cf := G • O k C g, k = 0,...,r. 
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Via the Kostant-Sekiguchi correspondence these G-orbits correspond to 
nilpotent i-C c -orbits Oq C , . . . , 0^ c in p + C p c . Under the Cayley type 
transform C G Int(gc) these Kc-orbits map to 

X k := C(Of c ) C n c = V c , k = 0,...,r. 

Since C(tc) = tc each X k is an Lq orbit in Vc, where Lc is a complexification 
of L, acting linearly on Vq. Under the identifications described above O k C 
Xk is a totally real submanifold. 

The Schrodinger model 

Rossi- Vergne [33] showed that the scalar type unitary highest weight rep- 
resentation corresponding to the Wallach point A G W can be realized on a 
space of L 2 -functions on the orbit Ok for A = k = 0, . . . , r — 1, and on 
the orbit O r = ft for A > (r — For convenience we put 0\ := Ok where 
either A = < k < r — 1, or k = r for A > (r — l)f . For each A G W 
there is a certain L-equivariant measure d/i^ on 0\ (see Section 11.21 for de- 
tails). Then the natural irreducible unitary representation of L ix exp(n) on 
L 2 (0\, d^x) extends to an irreducible unitary representation ir\ of G which 
is the scalar type highest weight representation belonging to the Wallach 
point A G W. The differential representation d-^ of q in this realization 
was computed in [22] and is given by differential operators on 0\ up to 
order 2 with polynomial coefficients. A special role is played by the second 
order vector-valued Bessel operator B\ which gives the differential action of 
n (see Section [1.61 for its definition). This operator was first introduced by 
Dib [IH] and studied further in [22|. 

Bessel functions on Jordan algebras 

For each A G W we construct Bessel functions J\{z,w) and I\(z,w) for 
z,w G X\ := Lc-0\ (see Sections 13.11 and [3 . 2|) . Both J\(z,w) an.&Z\(z,w) 
can be extended to functions on Vc X X\ which are holomorphic in z and 
antiholomorphic in w. They solve the differential equations 

(B\)zJ\(z,w) = -wj\{z,w), (B x ) z Z\(z,w) =wX x {z,w). 

For w = e the identity element and A > (r — 1)£ the corresponding one- 
variable functions J\{x) = J~\(x,e) and T\(x) = I\(x,e) on Q were first 
studied by Herz [21] for real symmetric matrices and later for the more 
general case in [TU1 [TTJ, [T21 151 EE] ■ We further define a K-Bessel function 
JC\(x) on 0\ for every A G W which solves the same differential equation 
as T\{x), but has a different asymptotic behaviour (see Section I33|) . These 
functions were introduced by Dib |IOj and studied further in [HI I15j. 
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The Fock space 

The closure of X^ = Oj^ c is an affine algebraic variety and hence the space 
C[A?fc] of regular functions on X^ equals the space V(Xk) of restrictions of 
holomorphic polynomials on Vc to X^. We construct measures on X^ in two 
steps. Fix A G W. First the L-equivariant measure d^\ on 0\ C X\ gives 
rise to an Lc-equivariant measure di>\ on X\ for every A G W. In the second 
step we extend the function 

uniquely to a positive density on X\ = Lc ■ 0\ which, under the identi- 
fication of X\ with a i^c-orbit, is invariant under the action of K. Then 
every polynomial F G V(X\) is square integrable on X\ with respect to the 
measure ui\dv\ and we let T\ be the closure of V(X\) with respect to the 
inner product 

(F, G)f x := const • / F(z)G(z)u> x (z)dv x (z), F,GeV(X x ). 

Theorem A (see Proposition 14.41 fc Theorems 14.151 15. 7p . For each A € 
W i/ie space is a Hilbert space of holomorphic functions on X\ with 
reproducing kernel 

(z w\ 
2' o")' z,w£X x . 

We further have the following intrinsic description: 

JF\ = {F\x x : F : Vc — > C holomorphic and 

f \F(z)\ 2 uJx(z)du x (z) <oo\. 

In Section 14.21 we prove that the inner product on T\ can also be ex- 
pressed as a Fischer type inner product involving the Bessel operator B\. 

Unitary action on the Fock space 

Complexifying the differential operators dTT\(X), X £ q, in the differential 
representation of the Schrodinger model one obtains a Lie algebra represen- 
tation dn9 of g on holomorphic functions on X\ by polynomial differential 
operators (see Section [2.51 for details). We define a g-module structure on 
V(X\) by composing this action with the Cayley type transform C G Int(gc): 

dpx(X) := d^(C(X)), X G g. 
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Theorem B (see Propositions B~9l KW\ & 14.171 fc Theorem f4. lip . For each 
A G W the Q-module (dp\,V(X\)) is an irreducible (g, K)-module which is 
infinitesimally unitary with respect to the inner product on F\ . It integrates 
to an irreducible unitary representation p\ ofG on F\ with associated variety 



where k G {0, . . . , r} is such that X\ = Xk- 

The group action of K in the Fock model is induced by the geometric 
action of Kc on the orbits X^ = 0^ c up to multiplication with a character 
(see Proposition 14. 12j) . Therefore the K-type decomposition of p\ equals 
the decomposition of the space of polynomials on 0^ c C p + with respect 
to the natural -Kc- ac ti°n which is essentially the Hua-Kostant-Schmid de- 
composition (see Proposition 14.81) . 

In Section 14.51 we use the Fock model to give explicit generators for the 
rings H(Xk) of differential operators on the singular affine algebraic varieties 
Xk ^ Vc, k = 0, . . . , r — 1, in terms of the Bessel operator B\. 

Although there are already known explicit realizations for highest weight 
representations on spaces of holomorphic functions our construction has a 
certain advantage. In the realization on holomorphic functions on the corre- 
sponding bounded symmetric domain G/K the group action is quite explicit, 
but the representation space as well as the inner product are defined in a 
rather abstract way, at least for the singular highest weight representations. 
In our Fock model the representation space is by Theorem lAl explicitly given 
by holomorphic functions on Vc which are square integrable with respect to 
a certain measure on the orbit X\. Although our group action is less explicit 
we still have a useful expression of the Lie algebra action in terms of the 
Bessel operators which allows e.g. the study of discrete branching laws as 
done in Theorem [El 

The Segal— Bargmann transform 

The unitary representations (7T\, L 2 (0\, dp\)) and (pai-^a) are isomorphic 
and we explicitly find the integral kernel of the intertwining operator which 
resembles the classical Segal-Bargmann transform ()0.ip . For this let tr G V£ 
denote the Jordan trace of the complex Jordan algebra Vc and recall the 
/-Bessel function Z\(z,w). 

Theorem C (see Proposition 15.11 fc Theorem I5.4p . Let A G W, then for 
ifi G L 2 (0\, dp\) the formula 



defines a function M\ifj G JF\. This gives a unitary isomorphism M\ : 
L 2 (0\, dpx) — > J~\ which intertwines the representations tt\ and p\. 



Ass(pa) = O 



k ' 




Z£X X 
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The £-finite vectors in the Fock model and in the bounded symmetric 
domain model have the same realization as polynomials on Vc and therefore 
also the intertwiner between these two models is worth studying. In Theorem 
15.101 we find an explicit formula for it in terms of its integral kernel. 

We remark that Brylinski-Kostant [7] construct a Fock space realiza- 
tion for minimal representations of non-Hermitian Lie groups as geometric 
quantization of the minimal nilpotent ifc-° r bit in (see also [5]). Apart 
from the fact that their cases are disjoint to ours they do not construct an 
intertwining operator between the known Schrodinger model and their Fock 
model such as our generalized Segal-Bargmann transform. Further their 
measure on the -ffc-° r bit is not positive whereas our measure is explicitly 
given in terms of the .ff-Bessel function and hence positive. Moreover, the 
Lie algebra action in their picture is given by pseudodifferential operators 
while in our model the Lie algebra acts by differential operators up to second 
order. 

Recently Achab pQ also constructed a Fock space realization for the min- 
imal representation of Hermitian groups. Her construction looks different 
from ours and it would be interesting to find a connection between her model 
and our model. 

The unitary inversion operator 

The explicit integral formula for the Segal-Bargmann transform can be 
used to find the integral kernel of the unitary inversion operator IA\ in 
the Schrodinger model. This operator IA\ is a unitary automorphism on 
L 2 (0\, dfi\) of order 2 (see Section [6] for its precise definition). The ac- 
tion of U\ together with the relatively simple action of a maximal parabolic 
subgroup generates the whole group action in the Schrodinger model. To 
describe its integral kernel recall the J-Bessel function J\{z,w). 

Theorem D (see Theorem I6.3|> . For each A € W the unitary inversion 
operator IA\ is given by 



In various special cases Theorem IDl was proved earlier. Ding-Gross [TT1 
Theorem 4.10 (iii)] and Faraut-Koranyi \lh\ Theorem XV. 4.1] established 
Theorem [D] for the relative holomorphic discrete series, i.e. A is contained 
in the continuous part of the Wallach set and is 'large enough'. (In fact 
Ding-Gross proved an analogous version of Theorem [D] for all holomorphic 
discrete series representations.) For q = su(k, k) and A in the discrete part 
of the Wallach set Theorem IDl was proved by Ding-Gross-Kunze-Richards 
[12\ Theorem 5.7]. For q = so(2,ra) and A the minimal non-zero discrete 
Wallach point Kobayashi-Mano gave two different proofs for this result (see 




$ e c 6 °°(0 A ). 
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[33t Theorem 6.1.1] and [344 Theorem 5.1.1]). The case of the minimal non- 
zero discrete Wallach point was systematically treated in [23] . The proof we 
present for the general case is quite simple, once the Bargmann transform 
is established, and works in the same way for all scalar type unitary highest 
weight representations. 

In Section 16.21 we use the integral kernel of U\ to construct explicit alge- 
braic and smooth Whittaker vectors in the Schrodinger model. 

Application to branching laws 

We demonstrate the use of our Fock model in one specific example, the 
branching so(2,n) \ so(2, m) ©so(n — to) of the minimal unitary highest 
weight representation of so(2,n). Note that the subalgebra f) := so(2,m) © 
so(n — m) of g = so(2, n) is symmetric and of holomorphic type, i.e. the cor- 
responding involution of g acts as the identity on the center of 6. Kobayashi 
[32 j proved that any highest weight representation of scalar type if restricted 
to a holomorphic type subalgebra is discretely decomposable and the decom- 
position is multiplicity free. He further determined the explicit decomposi- 
tion for representations in the relative holomorphic discrete series. We find 
the explicit decomposition for the non-zero discrete Wallach point by an 
easy computation in the Fock model. For j = m,n we denote by d7r B °^ 2 '^ 
the unitary highest weight representation of so (2, j) of scalar type with Wal- 
lach parameter A as constructed above. Further we let 'H k (M n ~ m ) be the 
irreducible representation of so(n — m) on the space of spherical harmonics 
on M. n ~ m of degree k. 

Theorem E (see Theorem 17. 2p . The unitary highest weight representation 
of q = 00(2, n) belonging to the minimal non-zero discrete Wallach point X = 
decomposes under restriction to the subalgebra f) = so(2,m)©so(n — m) 
as follows: 

oo 
fc=0 

We remark that for odd n the representation dTT S ^ 2 ' n ^ cannot be obtained 
via the theta correspondence and hence also the branching law cannot be 
obtained in this way. 

For even n this branching law was proved by Kobayashi-0rsted in [33| 
Theorem A] and, as pointed out to the author by B. 0rsted, their proof 
actually carries over also to the case of odd n, although it is not explicitly 
stated in their paper. However, the techniques they use are more involved 
than the computation we do in Section [7] to prove Theorem [El 

In the general case of a Hermitian Lie algebra ej Kobayashi-Oshima [36j 
recently classified all symmetric subalgebras of q such that the restriction 
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of the minimal scalar type unitary highest weight representation of g is 
discretely decomposable. In [33] the explicit decompositions in these cases 
are determined. 

Organization of the paper 

This paper is organized as follows: In Section Q] we recall the necessary back- 
ground of Euclidean Jordan algebras, their corresponding groups and nilpo- 
tent orbits. We also study polynomials and differential operators such as the 
Bessel operators on Jordan algebras. Section [2] describes three known models 
for unitary highest weight representations of scalar type, the bounded sym- 
metric domain model, the tube domain model and the Schrodinger model. 
We further investigate a natural complexification of the latter one. Section 
deals with Bessel functions on Jordan algebras. We introduce J-, I- and 
if-Bessel functions and discuss their properties. In Section 0] the Fock model 
for unitary highest weight representations of scalar type is constructed. We 
calculate the reproducing kernel of the Fock space and investigate rings 
of differential operators on the associated varieties. The Segal-Bargmann 
transform intertwining the Fock model and the Schrodinger model is intro- 
duced in Section [5j Here we also give a formula for the intertwiner between 
Fock model and bounded symmetric domain model. In Section [6] we use the 
Bargmann transform to obtain the integral kernel of the unitary inversion 
operator. We then describe Whittaker vectors in the Schrodinger model in 
terms of this integral kernel. Finally we use the Fock model in Section [7] 
to obtain in a very simple way the branching law for the smallest non-zero 
highest weight representation of so(2, n) restricted to so(2, m) © so(n — m). 

Acknowledgements: We thank B. 0rsted for helpful discussions on 
the topic of this paper. 

Notation: N = {1,2,3,...}, N = NU {0}, M x = R \ {0}, l + = {ie 
K : x > 0}, C x = C \ {0}, T = {z G C : \z\ = 1}, 

V*: algebraic dual of a vectorspace V, V: topological dual of a topological 
vectorspace V. 
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1 Preliminaries 



In this section we set up the notation and recall the construction of three 
known models for unitary highest weight representations of Hermitian Lie 
groups of tube type. For this we use the theory of Euclidean Jordan algebras 
which formalizes various aspects in a simple fashion. We mostly follow the 
notation of [15]. 

I. 1 Simple Euclidean Jordan algebras 

Let V be a simple Euclidean Jordan algebra of dimension n = dim V with 
unit element e. For x £ V we denote by L(x) G End(V) the multiplication 
by x and by 

P(x) = 2L{xf - L{x 2 ) 
the quadratic representation. Its polarized version is given by 

P(x, y) = L(x)L(y) + L(y)L(x) - L(xy), x, y G V. 

For x,y £ V we also define the box operator x\3y G End(y) by 

xDy := L(xy) + [L(x),L(y)] 

and note the identity (x\3y)z = P(x, z)y for x,y,z G V. 

The generic minimal polynomial of V provides the Jordan trace tr G V* 
and the Jordan determinant A, a homogeneous polynomial of degree r = 
rk V, the rank of V. Both can be expressed in terms of L{x) and P{x) (see 
[T5l Proposition III. 4.2]): 

Tr L(x) = -tr(x), DetPfs) = A(x) 2 ?, 

r 

for x G V. Here we use Tr and Det for the trace and determinant of a linear 
operator. Using the Jordan trace we define the trace form 

(x\y) := tx{xy), x,y G V, 

which is positive definite since V is Euclidean. Both L(x) and P(x) are 
symmetric operators with respect to the trace form (see |15L Proposition 

II. 4. 3]). For any orthonormal basis (e a ) a Q V with respect to the trace 
form we have the identity (see |15|. Exercise III. 6]) 

E4 = ?- (1-1) 

a 
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We use the trace form to normalize the Lebesgue measure dx on V . The 
trace form also defines a norm on V by 

\x\ := \J (x\x), 

Let c G V be an idempotent, i.e. c 2 = c. The only possible eigenvalues 
of the symmetric operator L(c) are 1, \ and 0. Denote by V(c, 1), V(c, \) 
and V(c, 0) the corresponding eigenspaces. The decomposition 

V = V(c,l) ®V(c,±) ®V(c,0) 

is called Peirce decomposition. The subspaces V(c, 1) and V(c, 0) are itself 
simple Euclidean Jordan algebras with identity elements c and e — c, respec- 
tively. An idempotent c G V is called primitive if it is non-zero and cannot 
be written as a nontrivial sum of two idempotents. Further, two idempo- 
tents ci, C2 £ V are called orthogonal if C1C2 = 0. (This implies (ci|c2) = 0.) 
A set of primitive orthogonal idempotents which add up to the identity e is 
called a Jordan frame. The cardinality of every Jordan frame is equal to the 
rank r of V. Further we say that an idempotent c G V has rank k if it can 
be written as the sum of k primitive orthogonal idempotents. From now on 
we fix a Jordan frame c\, . . . , c r . The symmetric operators L(ci), . . . , L{c r ) 
commute pairwise and therefore we obtain the mutual Peirce decomposition 

l<i<j<r 

where 



Vii = V( Cl ,l), l<i<r, 
V ij = V(c i) ±)nV(c j ,%), l<i<j<r. 

The spaces Vu are one-dimensional and spanned by q whereas the spaces 
Vij, 1 < i < j < r have a common dimension d. In particular, this implies 
the dimension formula 

n , / +\d 
- = 1 + ( r - 1)-. 

1.2 The structure group and its orbits 

The structure group Str(V) of V is defined to be the group of g G GL(V) 
such that 

P(gx) = gP(x)g*, x G V, 

where g& denotes the adjoint with respect to the trace form. This is by [T5l 
Exercise VIII. 5] equivalent to the existence of a scalar x{d) 6 ^ x such that 

Afoz) = xO/)A(x), x£V. 
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The map x '■ Str(V) — > M x defines a character of Str(V) which is on the 
identity component L := Str(V)o given by (see \15\ Proposition III. 4. 3]) 

X (g)=Bet(g)n, gel. 

Let 0(V) denote the orthogonal group of V with respect to the trace form. 
Then 

K L :=Lr\0(V) 

is a maximal compact subgroup of L, the corresponding Cartan involution 
being $(g) = g~^ . The group K L coincides with the group of Jordan algebra 
automorphisms, i.e. 

K L = {g G L : g(x • y) = gx • gy Vsc, y G V}. 
At the same time K L is the stabilizer subgroup of the unit element e, i.e. 

K L = {g G L : ge = e}. 

The Lie algebra [ = str(V) of L has the Cartan decomposition (see [I~5l 
Proposition VIII.2.6]) 

l = t l (Bp l , 

where ^ is the Lie algebra of K L consisting of all derivations of V and p' is 
the space of multiplication operators: 

£ l = {D G gl(V) : D{x ■ y) = Dx ■ y + x ■ Dy, Vac, y G V}, 
p l = L(V) = {L(x) : x G V}. 

The group L acts on V with finitely many orbits. The orbit Q := L ■ e 
is a symmetric cone and is isomorphic to the Riemannian symmetric space 
L/K L . TheLebes gue measure dx on V clearly restricts to fl. The following 
integral formula due to J.-L. Clerc Proposition 2.7] is used in Section \3. 31 
to calculate JT-Bessel functions on boundary orbits: 

Lemma 1.1. Let c G V be an idempotent of rank k and let fii and f^o denote 
the symmetric cones in the Euclidean subalgebras V(c, 1) and V(c, 0) , respec- 
tively. Further we let A°(x) be the determinant function of the Euclidean 
Jordan algebra V(c, 0). Then the following integral formula holds: 




A°(xo) dx dsi dxi. 
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Proof. Let $ : fii x V(c, 5) x f2 — > V be defined by $(xi, xi , xo) = 

exp(cdxi)(xi + xo). By [3HJ, Proposition 3] this map is a diffeomorphism 

2 

onto fi and is explicitly given by 

$(xi,Xl } Xo) = X\ + -c(xi(xiX )) + XiX + ^0 • 

9 V 9 9 9 ^ ^ 



Hence, in the Peirce decomposition V = V(c,l) © V{c,\) ® V(c,0) the 
Jacobian of <!> takes the form 



D$(xi,xi,x ) = L(a; )|y (C) i ) 
\ 

By [TS1 Propositions IV. 4.1 & IV. 4. 2] we have 

DetCLOcoVfci)) = 2- fc ('- fc ) d A°(x ) M 

and hence 

Det(D$(xi, xi , x )) = 2- fc ( r - fc ) d A°(x ) fcd . 

2 

Now the claimed integral formula follows from the transformation formula. 

□ 

The boundary d£l of the symmetric cone $7 has a stratification into lower 
dimensional orbits. In fact 

H = O U 01 U . . . U O r , (1.2) 

where 

0j := L • e 3 - with e& = ci + • • • + Cj. 

(We use the convention eo = here.) Note that O r = Each x G 0j has a 

polar decomposition x = ka with G A' L and a = 5Zj=i a « c «5 °i> • • • > a j > 
(see [15l Chapter VI. 2]). Assuming ai > . . . > aj > the element a is 
unique. We define complex powers of x = ka G 0& by x s := ka s , a s = 
Yjl=i a t c i f° r s £ C Note that for s G N this definition agrees with the 
usual definition of powers by the Jordan algebra multiplication. Since every 
k£ K L is a Jordan algebra automorphism the identity x s ■ x t = x s+t holds 
for s, t G C. 

The Gamma function of Q is for Re A > (r— 1)4 defined by the absolutely 
converging integral 



r n (A) := / e~ tr ^A(x) A -?dx 
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and is extended meromorphically to the whole complex plane by the identity 
(see [13 Theorem VILLI]) 



r n (A) = (2*)*? f[T (a - (j - 1)0 



3=1 

Using the Gamma function we define the Riesz distributions R\ G <S'(V") by 

(R x ,ip):=^-^ip(x)A(x) x ^dx, <peS(V). 

This integral converges for A > (r — 1)| and defines a tempered distribu- 
tion R\ G S'(V) which has an analytic continuation to all A G C (see [T5l 
Proposition VII. 2.1 & Theorem VII. 2. 2]). The distribution R\ is positive 
if and only if A belongs to the Wallach set (sometimes referred to as the 
Berezin-Wallach set, see [151 Theorem VII.3.1]) 

W = |o,~,...,(r-l)~ju f(r-l)-,oo 

Note that the Wallach set consists of a discrete part Wdisc an d a continuous 
part Wcont given by 



W dil 



{o,^...,(r-l)^}, W cont =((r-l)^,oo) 



For A G W C ont in the continuous part the distribution R\ is an L-equivariant 
measure dfi\ supported on Q which is absolutely continuous with respect 
to the Lebesgue measure dx. Hence the boundary dQ = O r _i is a set of 
measure zero. For A = fc| G Wdisc i n the discrete part the distribution R\ 
is an L-equivariant measure dfi\ supported on Ok for which dOk = Ok-i is 
a set of measure zero (see [T51 Proposition VII.2.3]). If we put 



for A = k%, < k < r - 1, 

1 2 ' 



Ox :-- 



for A > (r-l)l 



then we obtain measure spaces (0\, d/j,\) for A G W. The L-equivariant 
measures dfi\ transform by 

d^x(gx) = x(g) X dfj,(x), g G L. 

Note that with this normalization of the measures d(i\ the function ipo(x) = 
e -tr(x) a } wa y S n orm 1 in L 2 (0\, djix). In fact, for A > (r — 1)| we find 

|^o(x)| 2 d/i A (rE) = -^- / e- 2t ^A(x) x -Tdx 
1 n(Aj 



1 f e -^)A(x) x ^dx = l, 
1 n(Aj Jn 
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and the general case follows by analytic continuation. 

In terms of the polar decomposition of each orbit Ok we have the follow- 
ing integral formula (cf. [22l Proposition 1.8]) 

/ f(x) dfi x (x) = const • / / /(fce t )J A (t)dtd/c, (1.3) 

Jo x Jk l Jt 1 >...>t k 

where we use the notation t = (ti, . . . ,tk), 



E e '* c ». (1-4) 



i=l 



J X (t)=e^U^ J] simW^), (1.5) 

and k G {0, . . . , r} such that O x = O k . 

1.3 Conformal group and TKK algebra 

For a G V let n a be the translation by a, i.e. 

n a (x) = x + a, x £ V. 

Denote by N := {n a : a G V} the group of all translations. Further define 
the conformal inversion j by 

j(x) := for x G V invertible. 

The map j defines a rational transformation of V of order 2. The conformal 
group Co(V) of V is defined as the subgroup of all rational transformations 
of V generated by N, Str(V) and j: 



Co(y) := (iY,Str(F),j; 



grp- 



We let G := Co(F)o be its identity component. G is a connected simple 
Hermitian Lie group of tube type with trivial center and every such group 
occurs in this fashion (see Table [1] for a classification) . 

Conjugation with j defines a Cartan involution -d of G by 

■&{g) =j°9°j, 9 G G. 

Its restriction to L agrees with the previously introduced Cartan involution 
$(g) = g~^, g G L. Let K := G 15 be the corresponding maximal compact 
subgroup of K. 

The Lie algebra q of G is called Kantor-Koecher-Tits algebra. It acts 
on V by quadratic vectorfields of the form 

X(z) = u + Tz - P(z)v, zeV, 
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with u, v G V and Tel. For convenience we write X = (u, T,v) 6 g for 
short. In this notation the Lie bracket of Xj = (uj,Tj,Vj), j = 1, 2, is given 
by (see [15l Proposition X.5.8]) 

[X U X 2 } = (T 1 u 2 -T 2 u 1 ,[T ll T 2 } + 2(u 1 av 2 )-2(u 2 av 1 ),-T*V2 + T*v 1 ). 

This yields the grading 

= n © [ © n, 

where n = V is the Lie algebra of N and n = i?n. Thus, 
n = {(u,0,0) : u E V} 

acts via constant vectorfields, [ via linear vectorfields and 

n = {(0,0, u) : v £ V} 

by quadratic vectorfields. Note that the abelian subalgebras n and n together 
generate q as a Lie algebra. 

On q the Cartan involution i? acts via 

0(it, L> + L(a), u) = (-«, D - L(a), -u) 

and hence gives the Cartan decomposition q = t © p with 

t = {(«,£),-«) : n G V,D G 
p = {(u, L(a),u) : u,a £ V}. 

The Lie algebra t of K has a one-dimensional center (see [l2l Lemma 1.6.2]) 



and hence the universal cover K of K is the direct product of M. with a 
compact connected simply-connected semisimple group. Therefore the deck 
transformation group of the universal cover G of G is a finite extension of 



Under the action of K the space pc decomposes into two irreducible 
components 



Z(t) = RZ 



Z 




P: p' : P 



In fact, p^ 1 is the ±i eigenspace of ad(Zo) on pc and is explicitly given by 



P 



± 



{(u, ±2iL(u),u) : u G Fc}- 
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We set 



E := (e, 0, 0), H := (0, 2 id, 0), F := (0, 0, e). 

Then (E,F,H) forms an s^-triple in g. We define a Cayley type transform 
C G Int(gc) by the formula 

C := exp(-±iad(£))exp(-iad(F)). (1.6) 

It is then routine to check the following formulas: 

C(a,0,0) = (l,iL(a),a), (1.7) 
C(0,L(a) + D,0) = {i^D,-ia), (1.8) 
C(0 I 0,o) = (f,-tX(a),o). (1.9) 

From these formulas it is easy to see that the transform C induces a isomor- 
phisms 

tc He, («, D, -u) H> D + 2iL(u), 
p + Tic, (it, 2iL(u),u) i ^ (0, 0, 4m), 
p~ ^> nc, (u, —2iL(u),u) \-t (u, 0, 0). 

Since n and n together generate g as a Lie algebra it is now immediate that 
p + + p~ generates gc as a Lie algebra. Hence the real form {(a, L(b), a) : 
a,b G V} of p + + p~ generates g. 



v~ 


co(V) 


ste(V) 


n 


r 


d 


R 


st(2,R) 


R 


1 


1 





Sym(fc,R) (A; > 2) 


sp(/c,R) 


s[(fc,R) ©R 


\k{k + l) 


k 


1 


Herm(fc,C) (A: > 2) 


su(k, k) 


sl(fe, C) © R 


k 2 


k 


2 


Herm(V H) (* > 2) 


so*(Ak) 


su*(2/c) © R 


k(2k - 1) 


k 


4 


R 1 '* -1 (Jfe > 3) 


so(2,k) 


«o(l,Jfe- 1) © R 


k 


2 


k-2 


Herm(3, 0) 


e 7(-25) 


e 6( _ 26) © R 


27 


3 
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Table 1: Simple Euclidean Jordan algebras, corresponding groups and struc- 
ture constants 



1.4 Complexifications 

The complexification Vc of V is a complex Jordan algebra. The Jordan trace 
of Vc is given by the C-linear extension of the Jordan trace tr of V. The 
same holds for the trace form which is the C-bilinear extension of the trace 
form of V. By abuse of notation we use the same notation tr and (— |— ) for 
the complex objects. A norm on Vc is given by 

\z\ := y/(z\z), z G Vc- 
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Vc can also be considered as a real Jordan algebra and we write W = Vq 
for it. As real Jordan algebra W is simple since V is simple Euclidean. The 
trace form (— |— )w of W is of signature (n, n) and explicitly given by (see 
p| Lemma 1.2.3 (1)]) 

(z\w) w =2((Re(je)|Re(te)) - (Im(z)| Im(u;))) = 2Re(z\w), z,w € W. 

Note that (— |— )w is n °t C-linear. 

The identity component Lc := Str(Vc)o of the structure group of the 
complex Jordan algebra Vc is a natural complexification of L (see |15|, Propo- 
sition VIII.2.6]). As a real Lie group it is the same as Str(W)o, the identity 
component of the structure group of the real Jordan algebra W. Let U C Lc 
denote the analytic subgroup of Lc with Lie algebra 

u := t [ + ip l , 

then U is a maximal compact subgroup of Lc. The isomorphism C : he ~ * fc 
introduced in Section fl . 31 restricts to an isomorphism t — > u which integrates 
to a covering map r] : K — )■ U C Lq with differential 

drj(u, D, —u) =D + 2iL(u). 

Since uc = fc the subgroup C/ is totally real in Lc- 

For g G Lc we denote by 5* the adjoint of g with respect to the C- 
bilinear trace form (— |— ) of Vc- Since (— |— )w = 2Re(— |— ) this is also the 
adjoint with respect to the trace form (— |— )w of W. Further put 

9* ■■=§*, g£L c . 

Then g 1— > g~* defines a Cartan involution of Lc with corresponding maximal 
compact subgroup U . 

The determinant function Aw of the real Jordan algebra W is a poly- 
nomial of degree 2r. As in the Euclidean case an element g £ Str(W) is 
characterized by the property that there exists a scalar xw(d) £ such 
that 

Aw(gz) = xw{g)&w(z), z eW. 

This defines a real character \w '■ Lc — > M + . 

The group Lc has only one open orbit Lc • e (see [3T| Theorem 4.2]). 
Its boundary again has a stratification by lower-dimensional orbits. More 
precisely we have 

L c • e = X Q U Xi U . . . U X r , 

where Xk = Lc ■ &k (see [3T1 Theorem 4.2]). Note that Ok Q X^ is totally 
real and X\, is closed under conjugation. The closure Xf, of each Lc-orbit 
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is an affine algebraic subvariety of Vc ( see Theorem 2.9]). From 
Section 1.5.2] it is easy to deduce the following formula for the dimension of 

dime Xk = dim M O k = k + k(2r - k - (1.10) 

As in the real case every z G X\~ has a polar decomposition z = ua with 
u £ U and a = X^f=i a « c «' a i — • • • — a k > 0> the element a being unique 
with this property (see [151 Proposition X.3.2]). 

For A € W and k G {0, . . . , r} such that 0\ = we similarly put 
X\ := Affe. 

Proposition 1.2. For every A G W i/ie measure dv\ on X\ defined by 



f(z)dis x (z) = / / /(nx2)d/i A (x)dn (1.11) 

*A -/t/ -/OX 

is i/ie unique (up to scalar multiples) L^-equivariant measure on X\ which 
transforms by Xw- 

Proof. Except for the integral formula (jl.lip this is essentially [22], Proposi- 
tion 1.8]. By [221 Proposition 1.8] a x^-equivariant measure on X\ is given 
by 

/•"►// /(ueVfWdtdti, 

./£/ Jt 1 >...>t k 
where e is as in (|1. 41) . 

l<i<j<fc V w / V / 

and G {0, . . . , r} such that = A^. Using sinhx • coshx = i sinh2x we 
find 

Jf ( t ) = 2- fe ( fe - 1 )ieTE) =1 2% "Q sh V ( i i _i j) = 2 - fc ( fc - 1 )fj A (2t) 

i<i<j<k 

with J A (s) as in (fl~5|) . Now note that K L CU and (fee*) 3 = ke^ for k £ K L 
and hence 



/ / /(«e*)jf(t)dtd« 

J(7 Jt 1 >...>t k 

const- III f(uke t )J x (2t)dtdkdu 
7y Jifi Jti>...>t k 

const- / / / /(dh,^A(.)d.dtd B 

V[/ Ji^ Jsi>...>si. 



= const- / / f(uxz ) d/z^Oc) du, 
where we have used the integral formula (|1.3p for the last equality. □ 
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1.5 Nilpotent orbits and the Kostant— Sekiguchi correspon- 
dence 

Nilpotent coadjoint G-orbits 

We identify g* with g by means of the Killing form and view coadjoint 
G-orbits on g* as adjoint G-orbits on g. Further we use the embedding 
K ^ J, u 4 (u, 0, 0) to identify the L-orbits O k C V, k = 0, . . . , r, with 
L-orbits in n = V. Since n is nilpotent, the G-orbits 

Of := G ■ (e fc> 0,0) 

are nilpotent adjoint orbits in g. Since further O k = L ■ (efc,0, 0) we clearly 
have Ok Q O k . As usual we endow O k with the Kirillov-Kostant-Souriau 
symplectic form. The following result was proved in [22^ Theorem 2.9 (4)] 
for the minimal orbit 0\ . For the general case the author could not find the 
statement in the existing literature. 

Proposition 1.3. Ok Q Of is a Lagrangian subvariety. 

Proof. Since Ok Q n and n is abelian the symplectic form vanishes on O and 
it remains to show that dim Of = 2 dim Ok ■ By (jl.lOp we have 

dimO fc = k + k(2r -k - 1)^. 

To determine dim Of we note that 

= [(u, T, v), (e k , 0, 0)] = (Te k , -2e k Uv, 0) 

if and only if Tek = and v E V(e k , 0). Hence 

dimCf = dimOfc + dim(y/F(e fc ,0)) 

= dimOfc + dimy(efc, 1) + diml/(efc, |) 
= dimOfc + (k + k(k - 1)|) + k(r - k)d 

= 2dimO fc . □ 
Nilpotent A'c-orbits 

Via the Kostant-Sekiguchi correspondence the nilpotent adjoint G-orbits 
Of C g* correspond to nilpotent LTc-orbits 0^ c C p£. Identifying p* with 
p by means of the Killing form we view Ojf c as -K"c-° r t>its in pc = p + + p _ . 
Following [3S] we let (E k , F k , Hk) be the strictly normal sl2-triple in g given 
by 

E k :=(e k ,0,0), H k := (0, 2L(e fc ), 0), F k := (0, 0, e k ). 
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We form a new s^-triple (E%, F%, Hf) by putting 

Et := l(E k + F k + iH k ) = ^(e k ,2iL{e k ),e k ), 

Hi := i(E k - F k ) = i(e fcj 0, -e k ), 

F£ := -(E k + F k - iH k ) = -(e k , -2iL{e k ),e k ). 

Then E^ G p + and the Kc orbit 0^ c corresponding to O k is given by 

0*= = A :■ • Et 

Since p + is Kc-stable we have Ojf c C p + . 

If we use the embedding Vc > 0c> ^ (0, 0, u) to identify the Lc-orbits 
%k Q Vc, k = 0, . . . , r, with Lc-orbits in nc = Vc we obtain the following 
result: 

Proposition 1.4. The Cayley type transform C G Int(gc) is a bijection 
from the Kc- orbit 0^ c C p + onto the Lc- orbit X k C tv for every k G 
{0,...,r}. 

Proo/. Observe that C(L^) = (0,0,4e fc ). Since the Lie algebra homomor- 
phism C is a bijection from tc onto lc it gives a bijection between the orbits 
C»f c = K c ■ E% and Af fc = L c ■ (0, 0, Ae k ) which shows the claim. □ 



1.6 Differential operators 
General notation 

For a map / : X — > X on a (real or complex) vectorspace X we write 
Df(x) : X ^ X for its (real or complex) Jacobian at the point x £ X. For 
a complex-valued function j : I -> C we denote the directional derivative 
in the direction of a G X by <9 a g: 

5(3; + ta). 



t=0 



Bessel operators - the real case 

Let J be a real semisimple Jordan algebra with trace form (— |— )j. The 
gradient with respect to (— |— )j will be denoted by Jj- If (e a ) a C J is a 
basis of J with dual basis (e a ) a C J with respect to the trace form then the 
gradient is in coordinates x = ^ Q x a e a G J expressed as 

d_ x ^ a ^ 
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In terms of the directional derivative the gradient is characterized by the 
identity (a = d a for a G J. The Bessel operator B^ with parameter 

A G C is a vector-valued second order differential operator on J given by 

In coordinates it is given by 

a,/3 H a 

Denote by t the left-action of Str( J) on functions on J given by £(g)f(x) = 
f{g _1 x). Then the Bessel operator satisfies the following equivariance prop- 
erty: 

l^BiOg- 1 ) = g*Bi g G Str(J), (1.12) 

where g& denotes the adjoint with respect to the trace form of J. We further 
have the following product rule: 

H [f(x)g(x)] = B{f(x) ■ g(x) + 2P (^f, ?A x + f(x) ■ B{g{x). 



dx dx 

(1.13) 

For either J = V or J = W = Vc the operator B^ is tangential to the 
orbit Str(J)o ■ if A = ki , < k < r — 1, and hence defines a differential 
operator on this orbit (see [42(. Theorem 1.7.5]). On i^-invariant (resp. 
[/-invariant) functions on V (resp. W) we have the following formulas: 

Proposition 1.5. (1) ([El Theorem XV.2.7]) Let f G C°°(V) be K L - 
invariant. Then with F(a±, . . . , a r ) = f(a), a = YH=i a i c i> we have 

r 

BA/W = E(6lm«i.-,ar) Cl , 

i=l 

where 



I f d d 
a,-- a a 



a \ 



2 &i ~ a j \ ' d a i 3 daj 



(2) ([?, Proposition 4.1]) Let f G C°°(W) be U -invariant. Then with 
F(ai, ... ,a r ) = f(a), a = YJi=\ a i°i> we have 

r 

B^f{a) = Y J {K) l F{a l ,...,a r )c u 
i=i 



24 



where 



(e "'» , = i("4 + (2A " 1 " (r " 1)<i) i 

( 1 1 \ / 8 d 



— - + 



2 ^—f, V a, — a,- ^ a» + a,- / 1 * <9 a i ^ 9a 



For J = V we will write := for short. 
Bessel operators - the complex case 

We also need the Bessel operator of the complex Jordan algebra Vc- It is 
a holomorphic vector-valued differential operator on Vc which is defined in 
the same way as the Bessel operator of V. More precisely, if (e a ) a C Vc is 
a C-basis of Vc with dual basis (e Q ) a C Vq with respect to the trace form 
(— |— ) (which is C-bilinear) we define the gradient of Vc by 

d_ _ ^ d ^ 

dz dz a Q ' 

a 

where 



d I f d d 
i- 



dz a 2 \dx a dy a 

is the Wirtinger derivative and z = z a e a with z a = x a + iy a . The Bessel 
'a 



operator BY C of V"c is then defined by 



Ct,P H OL 



For a holomorphic function / on Vc we have 
B\{f\v) 



B v Cf 



V 



and hence B^ c is a natural complexification of the Bessel operator B\ of V . 



Therefore, by abuse of notation, we also abbreviate B\ = B 



1.7 Bounded symmetric domains of tube type 

Let := V + i£l C Vc be the tube domain associated with the symmet- 
ric cone 0. Each rational transformation g G G extends to a holomorphic 
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automorphism of Tq. This establishes an isomorphism between G and the 
identity component Aut(Xh)o of the group of all holomorphic automorphisms 
of Tq (see [T5l Theorem X.5.6]). Under this isomorphism the maximal com- 
pact subgroup KCG corresponds to the stabilizer subgroup of the element 
ie € Tq so that Tq = G/K. We identify an element g € G with its holomor- 
phic extension to Tq. 
Let 

p : Tq ->■ Vc, p(z) := (z - ie)(z + ie)" 1 , 

and define T> := p(Tq). Then p restricts to a holomorphic isomorphism 
p : Tq — > T> with inverse the Cayley transform (see [151 Theorem X.4.3]) 

c:V^Tq, c(w) = i(e + w)(e- io) -1 . (1.14) 

The open set T> C Vc is a bounded symmetric domain of tube type and 
every such domain occurs in this fashion. Clearly G is also isomorphic to the 
identity component Aut(P)o of the group of all holomorphic automorphisms 
of T> via the conjugation map 

a : G — > Aut(D)o, g h-> p o g o p" 1 = c" 1 o g o c. (1.15) 

The stabilizer subgroup of the origin £ D corresponds to the maximal 
compact subgroup K C G and also P = G/K. 

Denote by S C the Shilov boundary of T>. The group C/ acts tran- 
sitively on S and the stabilizer of e G S is equal to C f/ (see [T5] 
Proposition X.3.1 & Theorem X.4.6]). Hence E = U/K L is a compact 
symmetric space. Denote by dcr the normalized [/-invariant measure on 
E = U/K L . We write £ 2 (E) for L 2 (E, da) and denote the corresponding 
L 2 -inner product by (— , — )s- 

1.8 Polynomials 

In this Section we recall known properties for the space of polynomials on 
Principal minors and decompositions 

Let 'P(Vc) denote the space of holomorphic polynomials on Vc- The space 
'P(Vc) carries a natural representation £ of Lq given by 

l(g)p(z)=p(g- 1 z), g£Lc7peV ( Vc ). 

Since L and U are both real forms of the complex group Lq the decomposi- 
tions of V(Vc) into irreducible L- and [/-representations are the same. This 
decomposition can be described as follows: 
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For j G {l,...,r} we denote by Pj the orthogonal projection V — > 
V(ej,l). Let Ay( e )1 ) be the Jordan determinant of the Euclidean Jordan 
algebra V(ej, 1). We define a polynomial Aj on V by the formula 

Aj(x) : = A v(e . ;1) (P i x), x G V, 

and extend it to a holomorphic polynomial on Vq. The polynomials Ai, . . . , A r 
are called principal minors of V. Note that A r = A is the Jordan determi- 
nant of V. For m G Ng we write m > if mi > . . . > m r > 0. If m > we 
define a polynomial A m on Vc by 

A m (z) := A 1 (z) m ^ ■ ■ • A r _i(*) mr - 1 - rnr M*) ,np , * G Vb- 

The polynomials A m (z) are called generalized power functions. Define P m (Vc) 
to be the subspace of V(Vc) spanned by £(g)A m for g £ L. (Equivalently 
one can let g G U or g G Lc-) We write d m := dim'P m (Vc) for its dimension. 

Theorem 1.6 (Hua-Kostant-Schmid, see e.g. [T5l Theorem XL2.4]). For 

each m > the space f m (Vc) is an irreducible L-module and the space 
V(Vc) decomposes into irreducible L-modules as 

r(v c ) = ©p m (v c ). 

m>0 

The same spaces P m (Vc) occur in the decomposition of L 2 (E) into irre- 
ducible [/-representations. To make this precise let J7 + be the set of all 
m G 717 with rrt\ > . . . > m r . For m G we define another tuple 
m' := (mi — m r , . . . , m r _i — m r , 0). Then m' > and we define 

P m (S) := {A m 'p| s :pGP m ,(^c)}. 

If m > then 7 3 m (S) coincides with the space of restrictions of polynomials 
in V m (V c ) to S. 

Theorem 1.7 (Cartan-Helgason, see e.g. [Tot Theorem XII. 2. 2]). For eac/i 
m G Z!j_ i/te space V m (E>) is an irreducible unitary U -representation and the 
space L 2 {Ti) decomposes into the direct Hilbert space sum 

L 2 (S) = V m (E). 

meZ r + 

We now study the restriction of polynomials to the orbits C Vc and 
Ok Q V. Note that for m > and k G {0, . . . , r — 1} the condition nik+i = 
means mk+i = . . . = m r = 0. For convenience we also use this notation for 
k = r. In this case mk+i = should mean no restriction on m. 

Proposition 1.8 ( |24l Proposition 1.7 (vi) (a)]). For x £ we have 
Afc+i(x) = . . . = A r (x) = 0. In particular, A m vanishes on X^ iffmt+i ^ 0, 
m > 0. 
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For < k < r let V{X k ) and V m {X k ) be the images of V{V C ) and V m {V c ) 
under the restriction map C°°(Vc) — > C°°(X k ). Similar we define V(O k ) an d 
V m {Ok) to be the images of V(Vc) and V m (Vc) under the restriction map 
C°°(V C ) -> C°°(O fc ). 

Corollary 1.9. Xei /c G {0, ...,r} and m > 0. T/ien V m {X k ) (resp. 
V m {Ok)) is non-trivial if and only if m k +i = 0. In this case the restric- 
tion from Vc to X k (resp. O k ) induces an isomorphism "P m (Vc) = V m (X k ) 
(resp. VmiVc) = VmiOk)) of L -modules (resp. L^-modules) . In particular 
we have the following decompositions : 

V{X k )= V m (X k ), V{O k )= V m (O k ). 

m>0,m,fc +1 =0 m>0, mfe +1 =0 

For A G C and m > we define the Pochhammer symbol (A) m by 

(A) m :=n(A-(i-l)^) , 

i=l V / mi 

where (a) n = a(a + 1) • • • (a + n — 1) denotes the classical Pochhammer 
symbol for a G C, n G No- 

Lemma 1.10 ([TSl Lemma XI. 2. 3]). For p G 'Pm(V'c) and A G W we Ziaue 
/ e -(*lw>p(s) d/x A (x) = 2 rA (A) m A(y)^ A p(y- 1 ), y G a 

Spherical polynomials 

The representations 'Pm(Vc) of L (resp. U) are if ^-spherical. The if - 
spherical vectors in V m (Vc) are spanned by (see [151 Proposition XI. 3.1]) 

$ m (z) = / A m (fe)d/c, zG^c- 

The L 2 (£)-norm of these functions are given by (see [TBI Proposition XI. 4.1 (i)]) 

= m>0. (1.16) 

By \15\ Corollary XI. 3. 4] there exists a unique polynomial $> m (z,w) on 
Vc x Vc holomorphic in z and antiholomorphic in w such that 

$m{gz,w) = $ m (z,g*w), V^GLc, 
$m(x,x) = $ m (:r 2 ), VxGV. 

Lemma 1.11. (1) For all z G Vc we Ziaue 

$m(^,e) = & m (z). 
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(2) For x G V and y £ £1 we have 

(3) For z,w £ Vc we have 



®m(z,U)) = & m (w,z). 

Proof. (1) First let x £ f2. Recall the complex powers x s , s £ C, defined 
in Section fl.2l Then 



$m(a5,e) = <S> m (P(x± X 2,e) = $ m (^ , P( x i )e) 
= $ m 0,a^) = $ m (iE). 

Since both sides are holomorphic in x and C Vc is totally real, this 
also holds for x G Vc- 

(2) Let x £ V and y £ £1. Then by (1) we obtain 

&m(x,y) = <S> m (x,P(y^)e) = $ m (P(y5) x , e ) 
= *m(P(l/')a:)- 

Now both sides are continuous in y £ f2 and the claim follows. 

(3) First let x,y £ Q. Then by [TBI Lemma XIV. 1.2] there exists k £ K L 
such that P(yi)x = kP(xi)y. Using (2) and the i^-invariance of <& m 
we find that 

$mO,y) = $m{P{y^)x) = $ m (P(x^)y) 
= $m(y,x). 

Since <3? m (x,y) G R for x,y £ V, we obtain 



&m{z, w) = &m{w, z) V Z,W £ f2. 

Both sides are holomorphic in z and antiholomorphic in w and £1 C Vc 
is totally real. Hence, the formula also holds for z,w £ Vc- □ 

Lemma 1.12. //m > u>i£/i mfc+i 7^ 0, i/ien <I> m (,z,w) = for all z £ Vc 

and w £ 

Proof. Write w = he k with g £ Lq. Then 

$ m (z,w) = § m (z,ge k ) = $m(g*z,e k ). 

Therefore, it suffices to show that $ m ( — ,&k) = as a polynomial on Vc- 
Since $ m (z,w) is holomorphic in the first variable it suffices to show that 
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&m(— j Cfe) = as a polynomial on V and since f2 C V is open it is enough 
to prove &m(%, Cfc) = for x € 17. But for ifSlwe have by Lemma [1.1 II (2) 
and (3) 

<& m (x,e k ) = $ m (P(x5)e fc ). 

Now write = ge with g G L, then 

P(x5)e fc = P(ge)e k = gP(e)g*e k = gg*e k 

and hence P(x^)e k £ O k = L • e k . But by Corollary 11.91 we have 3> m |o fe = 
since m k+ \ ^ and the proof is complete. □ 

Example 1.13. On the rank 1 orbit X\ the spherical polynomials & m (z, w) 
are non-zero if and only if m = (m, 0, . . . , 0), m G No, and in this case 

a m m! 

(This can e.g. be derived from the expansion of tr(x) fc into the polynomials 
$ m (x), see [H3 Chapter XI. 5].) 

The Fischer inner product 

We equip V(Vc) with the Fischer inner product 



z=0 



where q(z) := q(z), z S Vc- The action of U on P(Vc) is unitary with 
respect to this inner product and hence the irreducible constituents V m (Vc) 
are pairwise orthogonal. Since U also acts unitarily on T'(Vc) with respect 
to the inner product on L 2 (T,) the two inner products are proportional on 
each irreducible constituent V m (Vc) (see [TS1 Corollary XI.4.2]): 

M = (~) P,qeV m (V c ). (1.17) 

Vr/m 

Denote by K m {z, w) the reproducing kernel of f m (Vc) with respect to the 
Fischer inner product. By [Tot Propositions XI. 3. 3 & XI. 4.1] we have 

A' m (z, w) = -^-$ m (z, w), z,w£ V C . 

Since by \15\ Proposition XI. 1.1] the completion of V(Vc) with respect to the 
Fischer inner product is a Hilbert space with reproducing kernel K (z, w) = 
e^ z \ w ) we obtain the following expansion: 



e (zW =^$ m (^), z,weV c . (1.18) 

m>0 1 
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Lemma 1.14. For m, n > and p G V m (Vc), q £ V n (Vc): 

/ p{uz)q(uw) du = $ m (z,w)(p, q)x V2,tflG % 

JU 

Proof. Using the reproducing kernel property and the Schur orthogonality 
relations we obtain 

p(uz)q{uw) du = / [p, K m ( — , uz)} [q, K n (— , uw)] du 
U JU 



[i{u- l )p, K m (-, z)} [e(u^)q, K»(-, w)} du 



d 



[ \p,q][K™(-,z),K"(-,w)}. 



With (fTTTTD and 



d 



[K m (-,z),K"(-,w)] = 8 mn K™(w,z) = 6 mn j^-$ m (z,w) 

\ r )m 

the claimed identity follows. □ 
Laguerre functions 

For m > the polynomial <£ m (e + x) is i^-invariant of degree |m| and can 
hence be written as a linear combination of the i^-invariant polynomials 
$ n (x) for |n| < |m|. Following |15|, Chapter XV. 4] we define the generalized 
binomial coefficients (*~) by the formula 

^m(e + rc)= (^) $n ^)' xeV - 

|n|<|m| W 

Using the generalized binomial coefficients we define the generalized Laguerre 
polynomials L^(x) by 

L x m (x) ■.= (\) m (Z)j^r^~ x)f xeV > (L19) 

|n|<|m| V ' { )n 

and the generalized Laguerre functions ^(x) by 

e^(x) :=e- tr ^L^(2x), x E V. (1.20) 

Both L^ 1 (x) and i^i 00 ) are -K^-invariant. Note that for A > (r— 1)| we have 
(^)n 7^ for all n > and hence L^ n {x) and ^(x) are defined on V. For 
x 6 Ok, k = 0, . . . , r — 1, we further have & n (x) = if n^+i / 0. Therefore 
the sum in 19[> reduces to a sum over n with nk+i = ... = n r =0. 
For such n we have (A) n 7^ for A = and hence the expression (|1.19p 
is defined. This gives Laguerre polynomials L^(x) and Laguerre functions 
t^aix) on ~Ol for each A G W. 

Properties of the generalized Laguerre functions have been studied in 

Hi- 
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2 Three different realizations of unitary highest 
weight representations of scalar type 

We describe three known realizations of highest weight representations of 
scalar type: the bounded symmetric domain model, the tube domain model 
and the Schrddinger model. We further discuss a complexification of the 
Schrodinger model which we use in Section 14.31 to construct yet another 
model, the Fock model. 

2.1 The Schrodinger model 

The highest weight representation of the universal cover G belonging to 
the Wallach point A G W can be realized on L 2 (0\, d/i\). We sketch the 
construction here (see e.g. [12] for details). First, there is a Lie algebra 
representation dir\ of q on C°°{0\) for every A G W given by 

d-K\(u, 0, 0) = i(u\x), 
d^ A (0,T,0) = d T # x + ^Tr(T*), 
d7r A (0,0,«) =i(v\B x ). 

Note that for A = /c|, k = 0, . . . , i — 1, the Bessel operator B\ is tangential 
to the orbit Ok and hence defines a differential operator acting on C°°(Ok) 
(see Section II. 6p . The representation dTT\ is further infinitesimally unitary 
with respect to the L 2 -inner product on L 2 (0\, dfj,\). 

The subrepresentation of ( dir\, C°°(0\)) generated by the function 

defines a (g, _ff)-module (dTt\,W x ) whose underlying vectorspace turns out 
to be (see e.g. [151 Proposition XIII. 3. 2]) 

W x = V(O x )e- tr{x \ 

The (g, -fT)-module (d7r\,W x ) integrates to an irreducible unitary repre- 
sentation (ir\, L 2 (0\, d/j,\)) of G. The minimal K-type is spanned by the 
function ipQ which is K-equivariant: 

nx(k)ip = £\(k)ip , k G K, 

where £\ : K — > T is the character of K with differential d£\(u,T, —u) = 
iAtr(u). This implies that the representation ir\ descends to a finite cover 
of G if and only if A G Q (which holds in particular for A G Wdisc)- For 
A > — 1 the representation tt\ belongs to the relative holomorphic discrete 
series of G. 
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2.2 The tube domain model 

For A G C consider the function 

T n xT n ^ C, 0, w) i-> A 

It is of positive type if and only if A G W (see [15|. Proposition XIII. 1.2 & 
Theorem XIII.2.4]). Denote by %\{Tq) the Hilbert space of holomorphic 
functions on Tq which has the function A(^p) _A as reproducing kernel. 
For A > 1 + (r — l)d this space coincides with the space of holomorphic 
functions F on Tq such that 

/ \F(z)\ 2 A(y) x ~^ dxdy < oo, 

where z = x + iy G Tq (see [T5], Chapter XIII. 1]). For every A G W there 
is an irreducible unitary representation 7r^ n of G on T-L\(Tq). Note that G 
acts on Tq by composition of the action of G = Aut(Th)o on with the 

- — ' rp 

covering map G — > G. Then the representation 7r A n is given by 
nf i (g)F(z) = f j^(g~ 1 ,z)F(g- 1 z), g £ G, z £ Tq, 

where the cocycle fJ^ n (g, z) is given by 

^(g,z)=Det(Dg(z))£, g€G,z£T n , 

the powers being well-defined on the universal cover G. 

The representations %\{Tq) and L 2 (0\, d/j,\) are isomorphic, the inter- 
twining operator being the Laplace transform (see |15|, Theorems XIII. 1.1 
& XIII.3.4]) 

C x : L 2 (O x , dfi x ) -> nl(T a ), Cx^(z) := [ e^ x ^(x) d/i A (x). (2.1) 

The Laplace transform Cx is a unitary isomorphism (up to a scalar) inter- 
twining the group actions. 

2.3 The bounded symmetric domain model 

The polynomial h(x) := A(e — x 2 ), x G V, is X L -invariant and therefore by 
[151 Corollary XI. 3. 4] there exists a unique polynomial h(z, w) holomorphic 
in z G Vq and antiholomorphic in w G Vc such that 

h(gz,w) = h(z,g*w), M g G L c , 

h(x,x) = h(x), Vx G V. 
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We also write h(z) := h(z,z) for z G Vc- Consider the powers h(z,w)~ for 
A € C as functions on T> x T>. Then h(z, w)~ x is positive definite on T> x T> if 
and only if A G W (see [T5l Theorem XIII.2.7]). The corresponding Hilbert 
space of holomorphic functions on T> with reproducing kernel h(z, w)~ x will 
be denoted by H\{V). For A > 1 + (r - l)d this space coincides with the 
space of holomorphic functions / on T> such that 

/ \f{w)\ 2 h{w) X ~^ dudv < oo, 
Jv 

where w = u + iv G T> (see |15t Proposition XIII. 1.4]). For each A G W 
there is an irreducible unitary representation tt^ of G on 7-L\(T>). To give an 
explicit formula recall the isomorphism a : G — > Aut(D)o defined in (|1.15p 
and view it as a covering map a : G — > Aut(X>)o- Then the representation 
tt^ is given by 

*x(9)fW = ^(g- 1 ,w)f(a(g)- 1 w), geG,weV, 
where the cocycle fi^(g,w) is given by 

l%{g,w) = Det(D(a(g))(w))£, g€G,w€V, 

the powers being well-defined on the universal cover G. 

The representations %\iT>) and 1~L\{Tq) are isomorphic, the intertwining 
operator being given by 

7A : n 2 x (T Q ) -> U\ip\ lx F(w) := A(e - w)- x F(c(w)), 

where c(w) is the Cayley transform defined in (|1.14p (see \15\ Proposition 
XIII. 1.3 & Theorem XIII. 3. 4]). The operator 7^ is unitary (up to a scalar) 
and intertwines the group actions. 

2.4 t-type decompositions 

In the bounded symmetric domain model the -ftT-type decomposition is very 
explicit. Let A G W and < k < r such that O x = O k . Then U\<(D) 
decomposes into the direct Hilbert space sum 

nl(p) = v m (v c ), 

m>0, m,fc +1 =0 

each summand V m (Vc) is irreducible under the action tt^(K) and on it the 
norm is given by (see [To], Theorem XIII.2.7]) 

\\pfni(v) = 7W*\\P\\l, P e ^{V c ). (2.2) 
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Further, the t'-spherical vector in each i^-type V m (Vc) is the spherical poly- 
nomial & m (z) which has norm 



l<i> II 2 - 



d m (A) m 

Correspondingly the underlying (g, i^)-module W A of the Schrodinger 
model L 2 (0\, d/i\) decomposes into iT-types 

^ A = , 

m>0, mfc_|_!=0 

where = C^ 1 o 7 A ~ 1 ('P m (Vc)). The f'-spherical vector in is given 
by the Laguerre function i^(x) defined in Section [1,81 (see [T5l Proposition 
XV. 4. 2]) which has norm (see pH Corollary XV. 4. 3 (i)]) 

WffX 1 1 2 _ (|)mWm , , 

K mllL2 (c , AidMA ) - ^ • ^-6) 



2.5 Complexification of the Schrodinger model 

The infinitesimal action d7rA in the Schrodinger model is given by second 
order differential operators on 0\ with polynomial coefficients. Hence the 
action can be extended to an action d7r^ of g on C°°(X\) by holomorphic 
differential operators. More precisely, let D be a differential operator on V 
with polynomial coefficients. Choose some basis e±, . . . ,e n of V and write 
x = X^j=i x j e j ^ V ■ Then D is in coordinates given by 

D = ^ Ca ( x )—^——^ 

with polynomials c a (x) of which only finitely many are non-zero. The coef- 
ficients extend uniquely to holomorphic polynomials c a {z) on Vc- We define 
the complexification D c of D to a holomorphic differential operator on Vc 
by 

Q\ a \ 

D C := Yl Ca ^dz^ ■■■dzZ"' 

where 

d I ( d , d 
dzj 2 \dxj dyj 

denotes the Wirtinger derivative for Zj = Xj + iyj in coordinates z = 
Y^=i z j e j ^ ^c- The operator D c is a complexification of D in the sense 
that for a holomorphic function / on Vc we have 

(D C f)\y = D(f\ V ). 
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Applying this procedure to dir\(X) for X G g we put 

dn^(X) := dTT X (Xf. 

It remains to show that these holomorphic differential operators are actually 
tangential to the orbit X\ and hence define an action d7r^ of g on C co {X\). 
For this we use the Schrodinger model of certain unipotent representations 
of the complex group Gc = Co(Vc) (the conformal group of the complex 
Jordan algebra Vc), viewed as a real Lie group. 

In [22l Proposition 2.14] the authors construct a representation dr\ of 
QCi viewed as a real Lie algebra, on C°°(X\) for each A G W. It is explicitly 
given by 

dr A (u,0,0) = i(u\z) w , 

dr A (0, T, 0) = 9 T#2 + ^ Tr w (T#), 
2 n 



dr x (0,0,v)=i(v\BY) 



w ■ 



By Tr^ we mean the real trace of an operator on the real vector space W. 
Note that dr\ does not act via holomorphic differential operators, but via 
real differential operators up to second order on X\. Further, its is shown 
in [221 Theorem 1.12] that the representation dr\ is infinitesimally unitary 
with respect to the inner product of L 2 (X\, dv\). 



We have the following result relating dr\ to the complexification d7r9 - 



Proposition 2.1. For X G g we have 

d^ x {X)= l -(dT X {X)-idr x (iX)). (2.4) 
In particular, for every X = (u,T,v) G g and all F,G G C°°(X\) we have 

dirf(u,T,v)F(z)-G(z)dv x (z) 

= [ F(z)- d^(u,-T,v)G{z)du x (z). 

JXx 

Remark 2.2. The formula (|2.4|) can be understood as an analog of the 
Wirtinger derivative 

d I f d d 
%■ 



dz 2 \dx dy / 

Proof of Proposition \2.1\ First note that since n and n together generate g 
as a Lie algebra it suffices to show ()2.4p for X G n and X G n. Further note 
that for z G Vc we have 

(a\z) = -((a\z)w — i(ia\z)w) , a G V. (2.5) 
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Then (|2^|) is immediate for X G n. Now let X = (0,0, a) G n. Let (e Q ) 
be any orthonormal basis of V with respect to the trace form (— |— ). Write 
x = Y2 a x a e a . We view W = Vc as a real Jordan algebra. The vectors 

U:=^e a and fa := 

constitute an R-basis of W. Its dual basis with respect to the trace form 
(-|-)w is given by (f a := f a ) a U (g a := -g a ) a . We write z = ^ Q ^ e a = 
Y^ a (a a fa + b a g a ) with a a , b a el and z a = -75 (a Q + Hence, 

did , 515 

and 



<9a a yjldx a db a y/2dy a 

with z Q = i a + x a ,y a G R. With (|2.5p we have 

i (dn(X) - idTj(iX)) = '- ((a|Sf) - ifialO) 



a,/3 



-(a|P(e a , e^z) + — 2J ~ — e a - i- — e r 



This shows (|2.4p for l£n and hence it follows for all X G 5. 
The stated integral formula now follows from (|2.4p using the fact that 
drx(X) is given by skew-adjoint real differential operators operators on 
L 2 (X\, dv\) with real coefficients if X = (0, T, 0) G and purely imagi- 
nary coefficients if X = (u,0,v) G g. □ 

Since dr\ restricts to an action on C°°(X\) by differential operators, 
the same is true for d-zr^ by the previous proposition. Therefore, d7r^ is 
a representation of q on C°°(X\) by holomorphic polynomial differential 
operators of order at most 2. 
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3 Bessel functions on Jordan algebras 



In this section we study J-, I- and K -Bessel functions on symmetric cones 
and their boundary orbits. These functions play a fundamental role in the 
study of Schrddinger and Fock models and the intertwining operators be- 
tween them. 



3.1 J-Bessel function 

For A € C with (A) m 7^ for all m > and z, w G Vc we put 

J X (Z,W) := J2 (-l) H fa wn *m(*,U/)- 
m>0 IjJml'Vm 

(This notation agrees with the one used in Chapter XV. 2].) One problem 
is that for a discrete Wallach point A = k = 0, . . . ,r — 1, we have 
(A) m = for all m > with rrik+i 7^ 0. However, by Lemma 11.121 we find 
that for z G Xk or w G X^: 

J x (z,w)= £ (-l)M *„(*,«;)■ 

m>0,m fc+1 =0 vr/ 

In this expression the coefficients are non-singular at A = k^ since (A) m 7^ 
for all m > with rrik+i = 0. Therefore we obtain for each A G W a J-Bessel 
function J\{z,w) on 1^ x ^. It only remains to show convergence of the 
defining series. 

Lemma 3.1. For A G W the series for J\{z, w) converges absolutely for all 
z G Vc and w G X\ and the following estimate holds: 

\Jx(z,w)\ < C(l + \z\ ■ |^|) li22 F ii e 2r v^ T H V z G V c ,weX~ x 

for some constant C > which depends only on the structure constants of 
V and A. 

Proof. Let z,w G Vc- Then by pT5|. Proposition X.3.2] there exist u G U and 
a = Y^ T j=i a j c j w hh a± > . . . > a r > such that w = ua = uP{az)e. With 
Lemma ll.lll (2) we find 

*m(«,«0 = ^ m (2,tiP(a5)e) = § m {P{a^)u- l z,e) 
= <S> m (P(a L 2)u- 1 z). 

Now suppose further that w G Affc, < k < r, and 77ifc+i = 0. Then 
a k+i = ■ ■ ■ = a r = and hence P(aa) projects onto V(efc, 1) C Xk- Thus 
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we find that P(a^)u 1 z E Xk an d again by [15} Proposition X.3.2] we can 
write 



P{a\)u~ l z = u'b, 



where u' G U and b = X^j=i ^jCj, bi > . . . > bk > 0. Now, by [T51 Theorem 
XII. 1.1 (i)] we obtain 

\<S>m(z,w)\ = \<S> m (u'b)\<b^---b™K 

We further have the following obvious inequalities (assuming rrik+i = for 
A = &!) 

dm < dimP H (y c ) = ( n + J m| i " ^ < d(l + Iml)"- 1 



' n \ 
, , m 



n^-(j-i)i) m] >H(i) mj =™\, 

3=1 3=1 
k r 

(A) m = J] (A - (i - > II ( A - ( fe - W 

3=1 3=1 

We abbreviate A' := A — (k — 1)| > 0. Putting things together gives 

\JX(Z,W)\ < Yl fgWU l $ m(-g,w)| 



n ( r )m(A)m 



^ (l + ^i)"' 1 •••(! + m k ) n -\ mi 



< 



j=l \m=0 



m!(A') mi ---(A') mfc 
m!(A')m " J j ' 



Now note that m6 m = (b-^)b m , so 

n-1 



1 + ^) ^(V;») 



where o-^i(/3; -2) denotes the hyper geometric function. For o-^i(/3; 2) we have 
the obvious identity 
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and hence (l + 6^) n o^i(A'; b) is a linear combination of functions of the 
type 



&* Fi(A' + k;b), 



k = 0, . . . , n — 1. 



Now by [3 equations (4.5.2) & (4.8.5)] the asymptotic behaviour of the 
hyper geometric function as z — > oo can be estimated by 



and hence we obtain 



n-l 



1 + 6 d&' ° F i( A '; 6 ) 



2„-2A'-l ^ 



<C 2 (1 + |6|) 2 — e 2Vb V6G[0,oo), 



the constant C2 > only depending on n and A'. Inserting this into the 
estimate for J\(z,w) gives 

\j x (z,w)\ < C 1 C$fl((l + \b j \) 2 ^e^) . 

Now note that 

|6| = \ u 'b\ = \P(a?)u~ x z\ 

< \\P{a?)\\ ■ \u~ 1 z\ = \\P(a*)\\ • \z\, 

where ||P(a2)|| 2 is the largest eigenvalue of P{a^)^ P{a^) = P{a^) 2 . Since 
P(aa) 2 acts on by a^aj, its largest eigenvalue is a\. Hence 

|6| < a\\z\ < \a\ ■ \z\ = \w\ ■ \z\. 

Altogether we finally obtain 



2n-2\'-l 



\jx{z, W )\<cic^ ma + ibj 

< CiC 2 i; (l + 6i + --- + 6 r 

r(2n-2A ; -l) 



r(2n-2\' -1) 



,2 y ^^b 1 +---+b r 



< dC^r s (1 + 1 

< C(l + \z\ • |-u;|) li22 F il e 2r V^ T H 

, r(2n-l) 

with C = CiC^r s > which gives the claim. 



rpn-aV-i) 2r ^ 



□ 



The estimate obtained in Lemma [3 .11 is not sharp, but suffices for most of 
our purposes. Recently, Nakahama |45j obtained a sharper estimate which 
we use in Section 16.21 to find explicit Whittaker vectors: 
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Proposition 3.2 (g^l Corollary 1.2]). For A G W and fe G N mi/i Re A + 
k > — — 1 i/iere exists a constant > suc/i i/iai 

\Jx(z\e)\ < C x , k (l + \z\\)e 2 \ lmz \\ z G ^, 

where \z\i = Y7j=i \ a j\> z = n Sj=i a j c j> u £ U, dj G M. 

Example 3.3. On the rank 1 orbit we have, using Lemma [1.12l and Example 

EH 

00 (—~\\ m 

m=0 



= T(\)J x _ 1 (2 y /{z\w)), z,weX u 

where J a (z) = (^)~ a J a (z) is the classical renormalized J-Bessel function 
which is an even entire function on C. 

The following proposition is clear with the results of Section 11,81 

Proposition 3.4. The J-Bessel function J x (z,w) has the following prop- 
erties: 



(1) J\(z,w) = J\(w,z) for z G Vc, w G X x , 

(2) J x (gz,w) = J\(z,g*w) for z G Vc, w G X\, g G L c . 

To prove the differential equation for w) we use the same method as 
Faraut-Koranyi [151 Theorem XV. 2. 6] for the one-variable J-Bessel function 
J(x,e). The first step is to calculate the Laplace transform as defined in 

Lemma 3.5. Fix w G X x and consider the function J\{—,w) on V given 
by x ^ J\(x,w). The Laplace transform of J x ( — ,w) is given by 

(C x Jx(-,w)) (z) = 2 rX A(-iz)- x e-^ w \ z G T n . 

Proof. Let < k < r be such that 0\ = O k . The we find, using Lemma fl.101 
and ([TIBjl : 

(C x J x (-,w))(z) 

^J x (x,w)d^ x (x) 



e 



m>0,ms,_i_i=0 



2 rX £ ( _ 1)H ^- A(-iz)-^ m (iz-\w) 
m>0 

2 rX A(-iz)~ x e~ i ^ W) . □ 
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Proposition 3.6. For A £ W i/ie function J x {z,w) solves the following 
differential equation: 

(B\)zJ\{z,w) = -wJx{z,w), z eVc,w G X\. 

Proof. First note that it suffices to show the differential equation for A > 
(r — 1)|, then the general case follows by analytic continuation. Further, it 
suffices to show the differential equation for z £ ft as J\{z, w) is holomorphic 
in z € Vc- Since the Laplace transform C x is injective on functions on Q the 
differential equation is equivalent to the identity 

Cx(B x w)) = -wC x J x (-,w). (3.1) 

Let z £ Tfi, then using the symmetry of the Bessel operator B\ we find 

(C x (B x J x (-,w)))(z) 

^ zlx) (B x ) x J x (x,w)dfi x (x) 



e 



o 



(B x ) x e^J x (x,w)d^ x (x) 
(P(iz)x + i\z)e l{zlx) J x {x, w) dfi x {x) 



i[P(z)^- + \z){{C x J x {-,w)){z). 







z 



Now by Lemma EES we have (C x J x {-, w)) (z) = 2 rX A(-iz)- x e-^ z 1 H. 
Using D(x~ 1 ) = —P(x)^ 1 and d y A(x) = A(x)(y\x~ 1 ) we find 

i \P{z)^ + *A [A(-w)- A e- i (*~ 1 l ,B) 

= i (-AP(z)^ 1 + iP{z)P{z)- 1 w + Xz) A(-iz)~ x e- i{z ^ w) 
= -wA(-«)- A e - i( ^ 1|¥) 

and (O) follows. □ 



3.2 /-Bessel function 

For A G W, z, G Vc and z G ^ or w G ^ we put 
l x (z, w) := Ja(-^, «0 = J\{z, -w) 

m > l r JmW m 

By definition the /-Bessel function I x (z,w) also satisfies the estimates in 
Proposition 13.11 and has the same properties as in Proposition 13.41 
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Example 3.7. By Example 13.31 it is immediate that on the rank 1 orbit we 
have 

i x (z,w) = r(A)7 A _ 1 (2 v / (7^)), z,wex u 

where I a (z) = (^)~ a I a (z) is the classical renormalized I-Bessel function 
which is an even entire function on C. 

Lemma 3.8. For A € W and y £ Q, z £ Vc 

f e- (xM l x {x,z)dfi x (x) = 2 rX A( y y x e( y ~ 1 W. 

Proof. This follows immediately from Lemma 13.51 □ 

Proposition 3.9. For A £ W i/ie function I\(z,w) solves the following 
differential equation: 

(B\) z l\(z, w) = wl\(z, w), z,w £ X\. 

Proof. Since I\(z,w) = J\(z, —w) this is equivalent to Proposition 13.61 □ 

3.3 J\~-Bessel function 

For A £ C and x £ Vt we put 

K x (x):= [ e- tr ^-^ u) A(u) x -^du (3.2) 
Jn 

= f e- tT ^-( x \ v ~^A(v)- x dv. (3.3) 
Jn 

Note that our normalization of the parameter A differs from the one used 
in [8] and [151 Chapter XVI. 3]. By [151 Proposition XVI. 3.1] these integrals 
converge for all A £ C and x £ Since the integrand is positive on Q we 
have K,\{x) > for x £ £1. To extend the if-Bessel function also to lower 
rank orbits we need the following result due to J.-L. Clerc [H Theoreme 4.1]: 

Proposition 3.10. Let c £ V be an idempotent of rank k. Let fii and 

fio be the symmetric cones in the Hermitian Jordan algebras V(c, 1) and 
V(c, 0), respectively. Further, let K\ be the K-Bessel function of Q±, Fq 
the Gamma function ofO,Q andriQ andr$ the dimension and rank ofV(c,0). 
Then for x\ £ 

K, x {x x ) = (27r) fc (- fe )f r Qo + A ) Kx{xi) - (3 - 4) 
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Proof. Denote by A 1 (x) and A°(x) the determinant functions of V(c, 1) and 
V(c, 0), respectively. Using Lemma ll. II we obtain 

£ A (xi)= / e- tr W-( a;i l u " 1 )A(u)- A du 



= 2 -k(r-k)d f f f e -tr(< i !.(«)) e -(x 1 |</ ) (n)- 1 ) 

A°(u ) kd A((i)(u))- x du dui duj 

2 

with 0(n) = expfcDni + no). Note that 

2 

(cDui) : V(c,l) -> 0, 

2 

(cDtii) : V(c, |) — > V(c, 1), x i-> c(nix), 
(cD«i) : V(c, 0) ->■ Wc, i), x (->■ nix, 

2 2 

and hence 

= (ui + -c(ui(uiu ))}) + (uiu )+ n , 

V 2 2 2 / 2 



6V(c,l) 6y(c, 3 ) 



whence 



tr(0(u)) = tr(ni) + -(X(n )ui |ni) + tr(« )- 
Further, we have 

<^(n) _1 = exp(cDui )~*(ui + no) 

= exp((e - c)D(-ni ))(n^f 1 + u^ 1 ] 

and by the same argument as above 

{ct>{u)- 1 \x 1 ) = {u^\x l ). 
Finally 

A((f)(u)) = A(exp(cDni)(ni +n )) 

= Det(exp(cDni )) « A(ni + no) 

2 

= exp (- Tt(cDui)) A 1 (ni)A°(n ) 
Vra 2 / 

and since Tr(cDni) = |tr(ni) = we obtain 
= AVi)A°(uo)- 
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e 



We now first calculate the integral over fii: 

tr(n 1 )-(x 1 K 1 ) A l^ i ^A dui = 

Next we calculate the integral over V(c, |): 

by the well-known integral formula for Gaussians. Further, 

Det(2L(no)| y(Ci i ) ) = A°(u ) fed 
by [15] Propositions IV. 4.1 & IV. 4. 2]. Finally we deal with the integral over 

[ e -w A o M f-A d?io=r ^ + ^_ A y 

Putting all together we obtain the claimed formula. □ 

This shows that for A near ki the Bessel function JC\(x) is defined for 
x G Ok and hence we obtain Bessel functions JC X on O x for A G W. Note 
that by (j3.4[) the function /Ca is positive on (Da- 



Example 3.11. (1) For V = R we have by [Ml formula 3.471 (9)] 
JC x {x) = 2K X -i(2Vx), x£Q = R + , 

where K a (z) = (|) a K a (z) is the classical renormalized K-Bessel 
function. 

(2) In the general case the Bessel function fC x is by Proposition 13.101 on 
the rank 1 orbit 0\ given by 



JC\(x) = const -/Ca(|x|ci) = const ■K X -i(2-\/\x\), x G 0\. 
Lemma 3.12. For A G W and m > we have 

[ p{x)TC x {x) d/j,\(x) = 2 rX T Q (^) m (X) mP (e), p G V m {V c ). 
In particular, for every N > we have 

[ (l + \x\) N IC x (x)dfi X (x) <oo. 
J 0\ 
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Proof. It suffices to show the claim for \> (r — The general case then 
follows by analytic continuation. For A > (r — 1)^ we have, using Lemma 

OS 

p(x))C\(x) dn\(x) 

p(x)K,\{x) A(x) x ~ t dx 

p(x)e- tl{u ~ 1) -^ u) A(u) x - 2 ?A(x) x -rdudx 



Ox 

2 rX 

2 r\ 




n Jn 



r n (A) 

= 2 rA (A) m / e-^^piu-^Aiu)-^ du 
Ju 

and under the coordinate change v = u~ l , dv = A(u)~~ du, this is 

= 2 rA (A) m I e-^p(v)dv 
= 2 rA (A) m r n (^)(^) m p(e), 



where we have again used Lemma 11.101 for the last equality. This shows the 
desired integral formula. 

For the second claim we observe that by the previous calculations every 
polynomial can be integrated against the positive measure K,\(x) dfj,\(x) 
since V(Vc) = © m >o^'m(^ / c) and hence the claim follows. □ 

Proposition 3.13. The function fC\{x) solves the following differential 
equation: 

BxfCx(x) = eK\(x). 

Proof. Differentiating under the integral we obtain 

BxK x {x) 

{P{-u)x + A(-n))e- tr(M " 1) - (a:|,t) A(n) A - v du 



/ - (p(u)^ + An) e-(*l«) • e-^^ACu)*-^ du 
W - (pW)e«j- + An) e-^l") • e^^A^-^ du 



(u ) x ~ — 
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Using d y P{x) = 2P(x,y), D(x~ 1 ) = -P(x)- 1 and d y A{x) = A(x)(y\x~ 1 ) 
we obtain 

= V f e-W u n2P{e a ,u)e a + P{u)e a tT{P{u)- l e a ) 

+ (A - ^){e a \u~ l )P{u)e a - \v}je- tr ^ A(u) x ~^ du 
= I e-^l") (2(^e^n + P(u)P(u)- l e 

+ (A - ^)P(u)^ 1 - Au)e- tr ( u ^A(u) A ^ du. 

By |LID this is 

= J e -( I W^ + e + (A- - \ u y- tT ^A(u) x -^ du 

= eJC\(x). □ 

Now let A G W. For z £ X\, z = ua with u £ U, a = Yli=i ^i c ii U — 0' 
we put 

((?)')■ 

We note that is positive on A^. 

Proposition 3.14. The function uj\(z) solves the following differential equa- 
tion: 

B\u\(z) = -uix(z), z £ X\. 

Proof. Recall the operators B\ and B^f acting on functions of r variables 
(see Proposition II. 5p . Let F(a±, . . . ,a r ) = K,\(a), a = Y^i=i a i c ii then F 
solves the system (B\) l F = F, i = 1, . . . , r by Propositions 11.51 and 13.131 
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i • ■ ■ i u n 



Put G{a u ...,a r ):= uj x (a) = F((f ) 2 , . . . , (f ) 2 ). Then 

(Sf)^!,...,^) 

lid 2 d 

= i (;. Sf + PA-i-(>--i)^ 

+ ^ y) ( — — + —J— ) Ui^- - a,— ) ) g(oi 

iffafd 2 i d \ en , , n a 

2 ~^ V a * - «i ai + ajj \ 2 9«i 2 da^J J \ \ 2 

a } - / /a, \ 2 9 2 ( . , ,,A 9 
+ I A — (r — 1)- 



2 



9a? \ 2 / 9a,- 



^ 2^ (f ) 2 -(f) 2 ^2/ 9a, V2/ doj J J \\ 2 J ''"'I 2 



■(^((f) 2 ,..,(f) 2 )=^(( an2 
: G(ai, . . . ,a r ) 



4 VAy V V 2 / ' V 2 7 / 4 V V 2 / 
4 



, . . . , 



Hence we obtain S^wa^) = | w a(^) f° r 2 = a. Since cja(^) is further U- 
invariant, we obtain with ()142p for z = ua with u £ U and a = Yll=i a i°i : 

BYu x (z) = u-*Bfu x (a) 

= u-u x (a) = -uj x (z) 

since vft = u" 1 for u G [/. Finally we use Proposition 12.11 to find for every 
aeV: 

i(a\B x )oj x {z) = d^(0,0, a)oo x (z) 

= 2 ( dr A(°>0> a VA(>) - «dr A (0,0,m)a; A (2;)) 
= 2* {( a \ B V)wU\{z) - i(ia\Bf) w uj x (z)) 

= 2* (( a l|)w ~ *(«o|f )w) ^aO) 
= i(a\l)ux(z). 

Since this holds for any a £ V we find B x u x (z) = |u>a(z) and the proof is 
complete. □ 
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Example 3.15. On the rank 1 orbit the function lo\ takes by Example 
IXTT1 (2) the form 

U\{z) = u\{\z\c{) = K\ ( ^— ci ) = const -K X -i(\z\). 
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4 A Fock model for unitary highest weight repre- 
sentations of scalar type 

In this section we construct a Fock space J-\ = J-(X X , uj x dv x ) of holomorphic 
functions on the orbit X x for every A G W, calculate its reproducing kernel 
and find a realization on JF X of the unitary highest weight representation 
corresponding to the Wallach point A. 

4.1 Construction of the Fock space 

Let A G W. Recall the positive function uj x G C°°(X X ) from Section [3731 We 
endow the space V{X X ) of polynomials on X\ with the L 2 -inner product of 
L 2 (X x ,u x dv x ): 

{F,G)r x :=— I F(z)G{z)u x (z)du x (z), F,GcV(X x ) (4.1) 
ca Jx x 

with c x = 2 3rA rn(^). This turns V(X X ) into a pre-Hilbert space. Its com- 
pletion T\ := F(X X , oj x du x ) will be called the Fock space on X x . 

It remains to show that the integral in (|4.ip converges. Using the polar- 
ization principle the following lemma suffices: 

Lemma 4.1. For F G V(Vc) we have 

[ \F(z)\ 2 u x (z)du x (z) <oo. 
Jx x 

Proof. Using the integral formula (jl.lip we obtain 

/ \F(z)\ 2 io x (z)du x (z) = [ [ |F(nx^)| 2 /C A (x)d^ A (x)d U . 
J X x JU J O x 

Now put 

p(x) := / \F(ux)\ 2 du. 
Ju 

Clearly p is a polynomial on V, so there are constants C\ > and N G N 

such that |p(x)| < Ci(l + jx])^. Now, every x £ Q has a decomposition 

x = ka with fc G if L and a = YJj=\ a j c ji a j — 0- m this decomposition the 

ii i i 

square root is of the form X2 = kaz with a 2 = X]j=i a j c j- Since the norm 

I — I on V is ET^-invariant we obtain 

\p{xH < C x (l + (ai + --- + a r )2") Af < Ci (l + \/f(a 2 + • • • + a 2 )^ * 
< C ly /r ( 1 + \x\2\ < dy/fl V2(l + \x\)2) 
= C 2 (1 + |X|)T 
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N 

with C2 = C\y/rT^ . Hence, we find 



\F{z)\'u x {z)dv x {z) = / p(x)IC x (x)d^ x (x) 



<C 2 [ (l + \x\)T!Cx(x)d/ix(x). 
Jo x 



The latter integral is finite by Lemma 13.121 and the proof is complete. □ 
We explicitly calculate the norms on the finite-dimensional subspaces 

V m (X X ). 

Proposition 4.2. Let m, n > and F G V m (X x ), G £ V n (X x ). Then 

(F,G)^=A^^) m (X) m (F,Gh. 

In particular the subspaces "Phi^a) Q 'Pi.Xx) are pairwise orthogonal with 
respect to the inner product {—, —)f x and for F = $ m we have 

||2 4H ( n )m ( A ) m 

Proof. Using the integral formula (jl.lip , Lemma 11.141 and Lemma 13.121 we 
obtain 

{F,G)r x = - [ F(z)G(z)uj x (z)dv x (z) 
c\ Jx x 



1 



F{ux^)G{ux^)uj x {x'^) dfi x (x) du 
= -(F,Gh[ $ m (x)/C A (f)d^ A (rE) 

CA J 0x 

CA Jo x 

= 4l m l(^) m (A) m (F,G) E . 

Since ||$ m |ls = Z~ by f|l- 16j) this finishes the proof. □ 

Remark 4.3. Comparing the norm on F x with the norm on the space 
U\{V) gives by fl£2J 



\\Ffr x = ^ m W m \\Ffuliv) ^ F e ^m(Vfc). 

If we denote by 0{X X ) the space of holomorphic functions on the complex 
manifold X x , we obtain the following result: 
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Proposition 4.4. 0(X x )PiL 2 (X x , lo x dv x ) is a closed subspace of L 2 (X X , oj\ dv x ) 
and the point evaluation 0(X X ) n L 2 (X x ,oj x di , x ) — > C, F \- > F(z), is con- 
tinuous for every z E X\. In particular, F x C 0{X\) n L 2 (X\, co x du x ) and 
the point evaluation F\ — > C, F h- > F{z) is continuous for every z 6 X\. 

Proof. This is a local statement and hence, we may transfer it with a chart 
map to an open domain U C C k . Here the measure u\du\ is absolutely 
continuous with respect to the Lebesgue measure dz and hence it suffices 
to show that C(C fc ) n L 2 (C k , dz) C L 2 (C fc , dz) is a closed subspace with 
continuous point evaluations. This is done e.g. in [20l Proposition 3.1 and 
Corollary 3.2]. □ 

The particular choice of the density ui\ yields the following result: 

Proposition 4.5. The adjoint of B\ on F\ is |. 

Proof. Let F and G be holomorphic functions on X\. Then by Proposition 
12.11 we know that 



/ B x F(z)G(z)L>x{z)dv x (z)= / F(z)B x G(z)u x (z) du x (z) 



The function G(z) is antiholomorphic and hence 

= f F{z)G{z)B x oj x {z)du x (z). 

By Proposition 13.141 we have B x uj x {z) = ju x (z) and therefore 

= [ F{z)-G{z)oj x {z)dv x {z). □ 
4.2 The Bessel-Fischer inner product 

We introduce another inner product on the space V(X\) of polynomials, the 
Bessel-Fischer inner product. For two polynomials p and q it is defined by 

\p,q]x ■= p(B\)q(4z)\ z=0 , 

where q(z) = q(z) is obtained by conjugating the coefficients of the polyno- 
mial q. A priori it is not even clear that this sesquilinear form is positive 
definite. 

Theorem 4.6. For p,q G V(X\) we have 

\p,q}x = (p,q}T x . (4.2) 
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The proof is similar to the proof of [5j Proposition 3.8] 



Proof. First note that for all p,q G V(X\) 



[(«lf )P,q]x = [P, {a\Bx)q]x 
(Hi)P,q)^x = (Pi {a\Bx)q)f x 



for a G Vc, 
for a G Ve- 



in fact, the second equation follows from Proposition [531 The first equation 
is immediate since the components (a\Bx), a G Vc, of the Bessel opera- 
tor form a commuting family of differential operators on Xx- Therefore, 
(a\Bx)p(Bx)q{4:z) = 4p(Bx){a\Bx)q{4:z) and the claim follows. 
To prove (|4.2p we proceed by induction on deg(q). First, if p = q = 1, the 
constant polynomial with value 1, it is clear that [p, q]x = 1 and by Proposi- 
tion !4.2l we also have (p, <?) j- A = 1. Thus, (|4.2p holds for deg(p) = deg(g) = 0. 
If now deg(p) is arbitrary and deg(q) = then (a\Bx)q = and hence 

[(a\f)P,q]x = [P, (a\B x )q]x = and 
((o|f)p,<7)^ = (p, {a\Bx)q)r x = 0. 

Therefore, (g^D holds if deg(g) = 0. We note that (02]) also holds if deg(p) = 
and deg(g) is arbitrary. In fact, 

\p,q}\ =p(0)q(0) = [q,p]x and (p,q)T x = (q,p)T x 

and (|4.2p follows from the previous considerations. Now assume ()4.2p holds 
for deg(g) < fc. For deg(q) < k + 1 we then have deg((a|£>A)(7) < k and 
hence, by the assumption 

Mi)p,q}\ = [p, (o|£a)<?]a = (P, (a\B x )q)T x = ((a,i)p,q)*x- 

This shows (j4.2|) for deg(g) < k + 1 and p(0) = 0, i.e. without constant 
term. But for constant p, i.e. deg(p) = 0, we have already seen that (|4.2|) 
holds and therefore the proof is complete. □ 

4.3 Unitary action on the Fock space 

In Section 12.51 we verified that for every A G W the complexification d-^ 
of the action d7rA defines a Lie algebra representation of q on C°°(Xx) by 
holomorphic polynomial differential operators in z. It is clear that this action 
preserves the subspace V(X\) of holomorphic polynomials. Using this we 
now construct an action of Q on V(X\) by composing the action d7r^ with 
the Cayley type transform C G Int(gc) introduced in Section [L~3l 

Definition 4.7. Let A G W. On V{Xx) we define a g-action dpx by 



dpx(X) := drt 



(C(X)) 
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Proposition 4.8. Let A G W and fc G {0, . . . , r} such that X\ = X^. The 
t-type decomposition of ( dp\, V(X\)) is given by 

oo 

m>0,mfc+i=0 

and in every t-type V m (X\) the space oft-fixed vectors is one- dimensional 
and spanned by the polynomial <& m ( z )- In particular (dp\,V(X\)) is an 
admissible -module. 

Proof. The Cayley type transform C : gc - > 0C maps tc to lc and t ( to itself. 
The action of lc in the complexified Schrodinger model d^ is induced by 
the action of Lc on the orbit X\ = L^-et up to multiplication by a character. 
Hence the £-type decomposition of V(X\) is the same as the decomposition 
into Lc-representations under the natural action I of Lq. Therefore the 
claimed decomposition is clear by Corollary 11.91 The action of t [ is induced 
by the natural action of K L and hence the unique (up to scalar) t'-invariant 
vector in the f-type V m (X\) is the unique (up to scalar) if L -invariant vector 
in "Pm^Xx) under the natural action I. This finishes the proof. □ 

Proposition 4.9. For each A G W the representation (dp\,T , (X\)) is an 
irreducible -module. 

Proof. Let m, n > be arbitrary and denote by U(q) the universal en- 
veloping algebra of g. Then it suffices to show that there exists an ele- 
ment of dp\(U(g)) which maps <& m to <E> n . Using Theorem 14.61 we have 
+ ||*m||^ = <£> m (Bx)$ m (4z)\ z=0 = 4l m l <S> m (Bx)$ m (z)\ z=0 . Since B x is a 
differential operator of Euler degree —1 the polynomial Q m (Bx)&m(z) is con- 
stant and by the previous observation it is non-zero. Note that dpx(U(&)) = 
dir^ (14(g)) contains multiplication by arbitrary polynomials and differential 
operators which are polynomials in the Bessel operator Bx- Hence, the op- 
erator $ n (z)$m(B\) G dpx(U(g)) maps $> m (z) to a non-zero multiple of 
$ n (z) and the claim follows. □ 

Proposition 4.10. The (g,t)-module (dpx,V(Xx)) is infinitesimally uni- 
tary with respect to the L? -inner product (—, —)f x - 

Proof. As remarked in Section [L3l the subspace {(a, L(b), a) : a,b G V} C 
generates g as a Lie algebra and therefore it suffices to show that the 
operators 

dpx(a,0,a) = ^i(a\4Bx + z) and dp A (0, 2L(a), 0) = ^(a\4B x - z) 

are skew-adjoint on V(Xx) with respect to the L 2 -inner product (— ,— )jr A . 
But this is clear by Proposition 14.51 □ 
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Theorem 4.11. The (g,t) -module (dp\,V(X\)) integrates to an irreducible 
unitary representation p\ of the universal cover G of G on T\. This repre- 
sentation factors to a finite cover of G if and only if A G Q. In particular it 
factors to a finite cover of G if X £ Wdi sc - 

Proof. By the previous results it only remains to check in which cases the 
minimal 6-type Vo(X\) = CI integrates to a finite cover. The 6-action on 1 
is given by 

d Px (a, D, -a)l = dvr?(0, D + 2iL(a), 0)1 = — Tr(2iL(a))l 

2n 

= zAtr(a)l. 

Therefore, the center Z(t) = R(e, 0, — e) acts by 
dp\(e, 0, — e)l = ir\l. 

In K we have e n ^ e '°'~ e ^ = 1 and hence, the claim follows. □ 

In the Fock model the action of the maximal compact subgroup is quite 
explicit. For this recall the group homomorphism n : K — > U C Lq with 
differential drj(u, D, —u) = D + 2iL(u) defined in Section [1.41 and the char- 
acter £\ : K — > T of K with differential d£\(u,T, —u) = iAtr(u) defined in 
Section 12.11 

Proposition 4.12. For k £ K we have 

Px (k)F(z) = Uk)F(7](k)*z), z e X\. 

Proof. The action of X = (u, D, —u) 6 is given by 

dp A (X) = dvr^O, d77(X),0) = d d „ w# +^Tr(dr ] (X)#) 
= dd v (x)# +»Atr(«) 
which implies the claim. □ 

4.4 The reproducing kernel 

We now calculate the reproducing kernel K\(z,w) of the Hilbert space F\. 
For this we first calculate the reproducing kernels K.™(z,w) on the finite- 
dimensional subspaces V m (X\) endowed with the inner product of T\. 

Lemma 4.13. Let X £ W and m > 0. If X = k = 0, . . . , r - 1, 

we additionally assume that mk+i = . . . = m r = 0. Then the following 
invariance property holds: 

Kf(gz,w)=Kf(z,g*w), z,w£X x ,g£L c . 
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Proof. Using Proposition 14.121 we find that for k G K and F G V^Xx) we 
have 



p x (k)F(z) = (p x (k)F,K^(-,z)) 



T. 



^p x (k~^{-,z))j 



= Uk- 1 )(F,KT( V (k)-#-,z))^ 

and on the other hand 

Px (k)F(z) = Uk)F( V (k)*z) 

= Uk)(F,K?(-, V (k)*z)) Tx . 

Since 7] : K ^ U \s surjective and u~ l = u* = u& as well as u G U for u G U 
we obtain 

K\™(uz,w) = K\ n (z,u*w), z,w£X x ,u£U. 

Now both sides are holomorphic in u G Lc and U C Lc is totally real. 
Hence the claim follows. □ 

Proposition 4.14. Let A G W and m > 0. If X = k^, k = 0, . . . ,r — 1, we 

additionally assume that = . . . = m r = 0. Then 

^M = 7XT^*™&Tl> z,^G* A . (4.3) 



Proof. First let A > (r - 1)|. By Lemma UTTBI the function K^(—,e) G 
T'W'^'a) is -fT^-invariant and hence there is a constant c™ such that K™(z, e) - 
c™$ m (-z)- Since 

1 = <Me) = ($ m ,^ A m (-,e))^ = cril^mll^, 

formula ()4.3p now follows with Proposition 14.21 Now, for A > (r — 1)| this 
implies the following identity: 

p(z) = \p,K?(-,z)] X = TTTT^Vt- • P(Bx) W $m(w,z)\ w=0 (4.4) 

for z G Vc and p G "P m (Mc)- The right hand side is clearly meromorphic 
in A with possible poles at the points where (A) m = 0. For A = k G 
{0, . . . , r — 1}, and m > with m,k+i = . . . = m r = we have (A) m ^ and 
hence the identity (|4.4p holds for such A and m by analytic continuation. 
Since the reproducing kernel of V m {X x ) is uniquely determined by (|4.4[) the 
claim now follows also for A = k^. □ 
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Theorem 4.15. The reproducing kernel M.\(z, w) of the Hilbert space J-\ is 
given by 

/ z w\ 

K\(z,w) = 1\ (-, — , z,w£X x . 



,2' 2 

Proof. Let < k < r be such that X\ = X k . By the previous result 
K™(z,w) is the reproducing kernel for the subspace V m (X\). Further we 
know by Proposition 14.21 that the spaces V m (Xx) are pairwise orthogonal. 
Therefore, by [46} Proposition 1.1.8], the sum 

£ kh,..>- E o (|5^-(I-?)-^(5-I 

m>0,m,fc +1 =0 m>0,m.fc +1 =0 T 

converges pointwise to the reproducing kernel M.\(z, w) of the direct Hilbert 
sum of all subspaces V m (X\) with m > 0, m^+i = 0. But this direct Hilbert 
sum is by definition T\ and the proof is complete. □ 

The following consequence is a standard result for reproducing kernel 
spaces and can e.g. be found in [351 page 9]. 

Corollary 4.16. For every F S T\ and every z £ X\ we have 

4.5 Rings of differential operators and associated varieties 

We recall the definition of the associated variety of an admissible represen- 
tation for the example (p\,T\). Let iUk{o))k&% denote the usual filtration 
of the universal enveloping algebra U{q) and form the corresponding graded 
algebra gr^(g) which is by the Poincare-Birkhoff-Witt theorem naturally 
isomorphic to the symmetric algebra S(qc)- The underlying (g, K)-module 
X x = V(X\) of the representation p\ carries an action of For m 6 No 

further let be the subspace of polynomials in V(X\) of degree < m. 
Then Xq = CI is X-invariant and generates X x as a ZY(g)-module. We 
further have 

d Px (U k {g))Xi = x£ +m , k,me N 0> 

i.e. the nitrations {Uk{o))k an d {X^) m are compatible. Thus the corre- 
sponding graded space 

oo 

&x x = Q)xi/xl_ 1 

m=0 

is a module over gr^(g). Consider the annihilator ideal 

-A, 



J A := Ann grW(g) (grX A ) C gr^( ) * S( Qc ) = C[ , 



CJ 
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Then the associated variety V(p\) of p\ is by definition the affine subvariety 
of g c consisting of the common zeros of J\. Since tc lives in degree 1 in 
grZY(g), but leaves each invariant, every element in V(p\) vanishes on Iq 
and we can view V(p\) as a subset of p c . Via the Killing form we identify 
p^. with p and view V(p\) as a subset of pc- It is well-known that the 
associated variety is a finite union of nilpotent i^c-orbits. 

Recall from Section [T3] the i^c-° r bits @k C Q P + which are isomorphic to 
the Lc-orbits Xk via the Cayley type transform C G Int(gc). The following 
result is due to A. Joseph [301 Theorem 7.14]: 

Proposition 4.17. Let A G W and < k < r such that X\ = X k . Then 

V( PX ) = ofz. 

Corollary 4.18. For A G W let k G {0, . . . ,r} such that X\ = X k . Then 
the Gelfand-Kirillov dimension of p\ is 2k + k(2r — k — l)d. 

Proof. It is well-known that the Gelfand-Kirillov dimension of an irreducible 
unitary representation equals twice the dimension of its associated variety 
in pp. Therefore the result follows from (jl.lOp . □ 

For an algebraic variety X over C denote by C[X] the ring of regular 
functions. Further let D(X) be the ring of algebraic differential operators on 
X. This subring of Endc(C[X]) can be defined inductively as follows: Let 
ID>o(X) := C[X] be the ring of multiplication operators and for m G N put 

D m (X) := {D G End c (C[X]) : [D, f] G D m _i(X)V/ G C[X]}. 

ThenD(X) = U meNo Bm(_X). _ 

Since the varieties X^ are affine it follows that C[Af^] = V{Xk). The 

representation dp\ acts on V(X\) by differential operators and hence it 
induces a map 

dp A : U(g) -> B(Xx). 

The following result is a qualitative version of [5UI Theorem 4.5]: 

Theorem 4.19. For A = k% G W di sc the map dp x : U{q) B(X~£) 
is surjective and induces an isomorphism U{$)lJk — ^{^k)> where J k = 
Ann w(0) (X A ). 

Proof. By [30l Theorem 4.5] the map dpA is surjective onto the space of C- 
endomorphisms of V{X\) = C[Xf.] which are locally finite under the diagonal 
action of p _ . Now p _ acts by multiplication with coordinate functions and 
hence the condition for D G Endc(C[Vtfc]) to be locally finite under the 
action of p~ is equivalent to the existence of N G N such that the iterated 
commutator [[. . . [D, fi(x)], f N -i(x)], /n(x)] = for all fi,...,f N G 
Cptfc]. This again is equivalent to D G ©(^fe) an d t ne proof is complete. □ 
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Remark 4.20. A quantitative version of Theorem 14.191 was obtained by 
Levasseur-Smith-Stafford [39] for the minimal orbit X\ , by Levasseur-Stafford 
[40| for classical g and finally by Joseph [30] for the general case. However, 
their version is less explicit and does not provide a geometric construction 
of the unitary structure. 

Corollary 4.21. For k = 1, . . . , r — 1 the ring B(Affc) of algebraic differential 
operators on the affine variety X^ is generated by the multiplications with 
regular functions in C[Xk\ and the Bessel operators (u\B\), u G Vc, for 
A = k^. 

Proof. By Theorem 14.191 the ring B(Affc) is generated by the constants and 
dpx(g). Note that in the decomposition gc = p~ ©tc©P + the subalgebra tc 
is generated by p + and p~ and hence O(Affc) is generated by the constants, 
d Px {p+) = {(u\B x ) :u€V c } and dp x (p~) = {(v\z) : v G ^ c }. □ 

Remark 4.22. Neither Theorem 14.191 nor Corollary 14.211 can hold for A G 
VV C ont resp. k = r since in this case X\ = X r = Vc and D(Vc) = C[x, ^] is 
a Weyl algebra (see [401 Lemma IV. 1.5]). 
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5 The Segal— Bar gmann transform 

For every A G W we explicitly construct an intertwining operator, the Segal- 
Bargmann transform, between the Schrodinger model (ir\, L 2 ((D\, d/J,\)) and 
the Fock model (p\,J~\) in terms of its integral kernel. 

5.1 Construction of the Segal— Bargmann transform 

For each A G W the Segal-Bargmann transform M\ is on ip G L 2 (0\, d[X\) 
defined by 

M x ij(z) := e"^ [ l x (z,x)e- tT ^^(x)d^ x (x), zGV c . (5.1) 
Jo x 

Recall the space 0(Vc) of holomorphic functions on Vc- 

Proposition 5.1. For tp G L 2 (0\, d(j,\) the integral in (|5.1|) converges uni- 
formly on bounded subsets and defines a function M x ip G 0(Vq). The Segal- 
Bargmann transform M\ is a continuous linear operator 

L 2 (O x , dux) ->• 0{V C ). 

Proof. Since the kernel function e~ 2 tT ^I\(z, x)e~ tr ^ is analytic in z, it 
suffices to show that its L 2 -norm in x has a uniform bound for \z\ < R, 
R > 0. By Lemma 13.11 there exists C > such that 

ft\(z,x)\ < C(l + \z\ ■ \x\) rJ2 ^e 2r "/^. 
Then for x G 0\, z G Vc with \z\ < R, we find 

^C'Cl + E-lxD^e^v^-N 

with C = C max| z |</j |e~ 2 tr (*0 | . This is certainly L 2 as a function of x G 0a 
with norm independent of z and the claim follows. □ 

Next, we show that M\ intertwines the action d~K\ on L 2 (0\, dfi\) with 
the action dp\ on T X - Recall the Cayley type transform C G Int(gc) intro- 
duced in Section [L3l 

Proposition 5.2. The following intertwining identity holds onL 2 (0\, d(i\)° 
B A o dTr x (X) = d7T^(C(X))oJB A , leg. (5.2) 
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Proof. As remarked in Section 11.31 the subspace {(a, L(b),a) : a,b G V} 
generates g as a Lie algebra. Hence it suffices to prove (|5.2p for the elements 
(a,±2iL(a),a), oeF. We show jOJ) for X = (a, -2iL(o), a), a G V, the 
proof for (a, +2iL(a), a) works similarly. For X = (a, — 2iL(a), a) we have 
C(X) = (a, 0, 0) and hence 

= i(a\z)e- 1 2 tv{z) [ l x (z,x)e- tl{x) ^(x)dfi X {x). 
Jo x 

By Proposition 13.91 we have zl x (z,x) = (B x ) x I x (z,x). Further the Bessel 
operator B\ is symmetric on L 2 (0\, d/j, x ) and we obtain 



le 



te 



l tr W / (a\B x ) x l x (z,x)e- t ^^(x)d f , x (x) 



W*) f X x {z,x){a\B x )\e-^ x ^(x)\ dfi\(x). 



By the product rule (|1 . 13[) for the Bessel operator we obtain 



By 



■ tr(x 



B x e~ tr{x) ■ i){x) + 2P (JL e - tT ( x \ x + e~ tT ^ ■ B, 

(x - Xe)e- tl(x U(x) - 2e~ tlix) x ■ ^ + e - tl(x) B x iP(x) 

ox 



ip{x) 



since 



B Ae -tr(x) = (p(_ e ) x _ Ae)e- tr W = (x - Ae)e" tr(:r) 
dx 

Hence we have 



and 



-e • e 



-tr(x) 



(a\B> 



- tr(x 



^tp(x) 



= e- tr ^ [(a\x)^(x) - \tr(a)iP(x) - 2d ax ^{x) + B x ^(x)] . 
Inserting this into our calculation above we find 
(d^(C(X))oMMz) 



A o i(a\x) — 2i 



d L{a)x + r ^Tr(L{a)) 
l x o dvr A (X))V(z). 



+ i(a\B x ))^(z) 



□ 
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To conclude that B A is an isomorphism L 2 (0\, dpx) — > J-\, we show that 
it maps the underlying (g, t)-module W x C L 2 ((D\, dp\) to the (g, 6)-module 
V(X\). In order to do so we show that the function 

tf := B A Vo G O(Fc) 

is Lc-invariant. In fact, the function ip$ is i^-equivariant by the character 
£a (see Section l2.ip . By Proposition 15.21 the same has to be true for ^q. 
But by Proposition 14. 121 this implies that is invariant under r](K)# = U. 
Now U is a real form of Lc an d the action of Lq on *$>q is holomorphic, 
whence nas to be Lc-invariant. Therefore it has to be constant on every 
Lc-orbit. Since is holomorphic on Vc and Vc decomposes into finitely 
many Lc-° r bits, it follows that is constant on Vc- It remains to show 
that is non-zero. 

Proposition 5.3. ^o(O) = 1 and hence \I/o = BaV'o = 1- 
Proof. Since Z\(0, x) = 1 we have 

*o(0) = / e- 2 "Wd W (x) = Uo\\l H o M = 1 

as shown in Section 11.21 □ 

Theorem 5.4. M\ is a unitary isomorphism L 2 ((D\, dfix) ~~ ^ 3~\ intertwin- 
ing the actions tt\ and p\. 

Proof. Since W x = L 2 (O x , d^ A )t is generated by ipo, V{X\) = (J\)t is 
generated by = 1> BaV'o = & n d B A intertwines the actions d7r\ and 
dpx, it has to map the irreducible (g, 5)-module W x into the irreducible (g, £)- 
module V(X\) and is therefore, thanks to Schur's Lemma, an isomorphism 
between the underlying (g, t)-modules. It only remains to show that M\ is 
isometric between W x and V(X\), then the statement follows since W x C 
L 2 (O x , d/j, x ) and V{X\) C F x are dense. 

Both W x and V(X\) are irreducible infinitesimally unitary (g, t)-modules 
and hence M\ is a scalar multiple of a unitary operator. Since further 

||BaV>o||j- a = U\\T X = 1 = ||^o|k 2 (O x ,dAi A ) 

the operator B A itself has to be unitary. □ 

Remark 5.5. The Segal-Bargmann transform can also be obtained via a 
restriction principle (see [281 147j for other instances of this principle). The 
formula 1Z\F(x) = e~^ tr ^F(x) defines an operator V(X X ) — > L 2 ((D\, dfi x ) 
and hence we obtain a densely defined unbounded operator 1Z\ : J-\ — > 
L 2 (0\, dfi\). We consider its adjoint 1Z\ : L 2 ((D\, dfj,\) — > T\ as a densely 
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defined unbounded operator. One can show that the Segal-Bargmann trans- 
form appears as the unitary part in the polar decomposition of the operator 



For the case of the minimal discrete Wallach point this is done in [231 Propo- 
sition 5.5]. 



Corollary 5.6. The inverse Segal-Bargmann transform is given by 

-tr(x) 

ca Jx x 

Proof. Since the Segal-Bargmann transform is a unitary operator we have 



; 1 F(x) = - / l x (x,z)e- 1 2 t ^F(z)u x (z)d^(z), F G F x 

C A Jx, 



(B- 1 F,^) i2(OA , dMA) = (FM) Tx = - f F(z)^{z)u x {z)dv x {z) 

ca Jx x 

= -f [ F(z)e-^%^e-^J{^^ x {x)uj x {z)dv x (z) 

CA Jx x Jo x 

= ±[ f e -trix) T e -h^)T x (x,z)F{z)uj x (z)du x (z))i^dfi X (x) 

CA Jo x V JX X J 

which implies the claim. □ 

We now use the Segal-Bargmann transform to obtain an intrinsic de- 
scription of the Fock space. 



€0(V C ),[ \F(z)\ 2 io x (z)dv x (z) 



Theorem 5.7. 

T x = ■{ F\x x ■ F G 0(V C ), j x \F{z)\ z u x {z) du x (z) < oo 

Proof. The Segal-Bargmann transform is an isomorphism B> : L 2 (O x , dfi x ) - 
J- x and hence by Proposition 15.11 every function in F x extends to a holomor- 
phic function on Vc- This shows the inclusion C. 
For the other inclusion let F G 0(Vc) such that 

F(z)\ 2 u x {z)dv x {z) < oo. 

We expand F into a power series which we can arrange as 
F = ^ Pm with p m G V m (Vc) 

m>0 

by Theorem 11.61 Since p m \x x = for mk+i ^ by Corollary 11.91 we may 
assume F = ^ m >o m k+1 =oP m ^ or * ne study of F\x x . This series converges 
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uniformly on compact subsets. We show that this series also converges in 
L 2 (X\, uj\ Av\). For R > let 

X? := {z G X\ : \z\ < R}, 
Of := {x G 0\ : \x\ < R}. 

Note that and are compact and hence integration over these sets 
commutes with taking the limit F = X^ m >o m k+1 =oPm- With ([l.lip we find 



o> / \F(z)\ 2 u x (z)du x {z) 
Jx x 



lim / \F(z)\ z uj x (z)Au x {z) 
R-^°° Jx? 



-Urn Y. / /M-i)^d^(xi)d, iW . 

R-^-oo *■ — ' //nil /rr 

m,n>0, J U X JU 

m k+l =n k+1 =0 

By Theorem 1 1 . 71 and Lemma [1.141 the integrals over U for m 7^ n vanish and 
we obtain 



lim V] / |p m (z)| 2 WA(2;) dz> A (^) 

»00 ' — ' / yR 



m>0,m fe+1 =0 A 

= ^ HP«»lli 2 (# A ,u;*dioO 
m>0, mfc_|_i=0 

which shows convergence in L (X\,(jj\dv\). Since F\ is the closure of the 
space of polynomials with respect to the norm of L 2 (X\,lj\ dv\) it is clear 
that F\x x G T\ which shows the other inclusion. □ 

This intrinsic description leads to the following conjecture which was 
proved in |23[ Theorem 2.26] for the minimal orbit, i.e. A G W with X\ = X\\ 

Conjecture 5.8. 

T\ = \f^ 0(X x ) : jf \F(z)\ 2 u> x (z)du x (z) < 00 J . 

Recall the Laguerre functions introduced in Section [1.81 

Proposition 5.9. Let A G W and k G {0, . . . ,r} such that 0\ = Ok- Then 
for every m > with mk+i = we have 

1)H 



2|m| 
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Proof. £ m is the unique (up to scalar multiples) ^'-invariant vector in the t- 
type W m C L 2 (0\, d[i\) whereas $ m is the unique (up to scalar multiples) 
r-invariant vector in the £-type V m {X\) C F\. Hence 

M x £ m = const -$ m . 

To find the constant we evaluate B^" 1 $ m (x) and £ m (x) at x = 0. First 
observe that 



observe that 

4(0) = L m (0) = (A) 



m ■ 



Next we calculate B^ 1 $ m (0) using Corollary 15.61 
c\ J Xx 

By (|1.18p the function e tT ^ has the following expansion: 

m>0 r 

Inserting this yields 

n > Z W'V ° X J *\ 

= E(- 1 ) H 2i=fer<*-'*->^ 

n>n IrJn 



n>0 

f ]W Jm_ 4l m l(2 )m ( A ) m 

1 j 2M(2) m 
= (-l)M 2 M(A) m , 



where we have used Proposition 14.21 □ 



5.2 Intertwiner to the bounded symmetric domain model 

The Fock space T\ and the bounded symmetric domain model 1i\{T>) have 
(as vectorspaces) the same underlying (g, i^)-module, namely 

Pm(tfc), 
m>0, rrtfc +1 =0 

where k G {0, . . . , r} is such that X\ = Xk- However, inner product and 
group action differ. In Remark 14.31 we already compared the norms in the 
two models. To gain a better understanding of the relation between the two 
models we investigate the intertwiner between them. 
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For this we compose the inverse Segal-Bargmann transform B^" 1 with the 
Laplace transform C x and the pullback of the Cayley transform 7^ to obtain 
an intertwining operator between the Fock space picture and the realization 
on functions on the bounded symmetric domain. (For the definition of C\ 
and 7a see Sections 12.21 and 12.30 Let 

A X := 7A ° £ A ° B- 1 : T X ^H 2 X (V). 
The operator A x intertwines the actions p x and ttV. 
Theorem 5.10. For F £ T\ we have 

A x F(z) = — [ e~^ z ^F{w)uj x {w)du x (w), z£V. 

CA Jx x 

Proof. Using Lemma 13.81 we obtain 

A x F(z) = lx oC x o M^F(z) = A(e - z)~ x C x o B^F(c(*)) 
= A(e - z)~ x [ e^^B^Fiu) dfi x {u) 

e i(c(*)|«) e -tr(«)x A ( Uj w ) e ~\ *&)F(w) 



A(e-z)- 



CA Jo x Jx x 

u) X (w)du x (w)dfj, x (u) 

= A(e-zT^ f (f e -Me-*c(*)) x , )U ,)<w„ )) e -h<^)F( w) 

CA Jx x \Jo x ) 

u x (w)dv x (w) 

= r X A(e- Z )- X f A(e _ ic(z)r A e ((e^ C (,))- Ne -|tr(W )i?W 
CA Jx x 

uj x (w)dv x (w). 

Now 

e - ic(z) = e + (e + z)(e - z)~ l = 2(e - z)' 1 
and therefore 

A(e - ic{z)Y x = 2~ rA A(e - z) x . 
This also implies that 

((e - ic(z))- 1 \w) = i tr(uJ) - ^(z\w) 
so that altogether we obtain 

A x F(z) = — [ e-^ w) F(w)u x (w)dv x (w). □ 

CA JX X 



66 



Corollary 5.11. The operator A\ acts on V m {Yc) by the scalar — 
Proof. Using the expansion (|1.18p and Proposition 14.141 we obtain 

eW = ^(W, ffl ). 

n>0 

Hence for F £ V m (Vc) we find 

A x F(z) = J2^UF,K^(-,-lz)) Tx 

n>0 

= (A) m F(-I.) = ( - 1 ffl m % 
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6 The unitary inversion operator 

Using the Segal-Bargmann transform we now calculate the integral kernel 
of the unitary inversion operator IA\. The unitary inversion operator is 
essentially given by the action ir\(j) of the inversion element (see \22\ Section 
3.3]) 

j = expg ^(e,0,-e)J G G. 
More precisely, we put 
U x ■= e- lnr T7rxQ). 

The operator Ux is unitary on L 2 (0 A , dfix) of order 2, i.e. U\ = id (see [22"1 
Proposition 3.17 (1)]). We also study Whittaker vectors in the Schrodinger 
model. They can be derived from the explicit expression of the integral 
kernel of Ux ■ 

6.1 The integral kernel of Ux 

Recall the underlying (g,t)-module W x = V(Ox)e~ tT ^ of the representa- 
tion (ttx,L 2 (Ox, dfix))- 

Proposition 6.1. For every if) G W x the integral 

T\ip(z) := 2~ rX [ J x (z, x)i>{x) d/i A (x), z G V c , 

converges uniformly on corn-pact subsets of Vc- This defines a linear operator 
Tx:W x ^C(O x ). 

Proof. Let ip(x) = p(x)e~ tr ^ G P(C A )e~ tr(a;) = W x . Then \p(x)\ < d(l + 
\x\) N for some constants C\,N > 0. Further, by Lemma 13.11 we have 

\Jx(z,x)\<C 2 (l + \z\-\x\) rJ ^e 2r V^\ y xe o x ,z£Vc 

for a constant C 2 > 0. Hence, for \z\ < R, R > 1, we obtain 

/ \Jx(z,x)if)(x)\dnx(x) 

< dC 2 [ (1 + R\ x \)N+ r -^^ e 2rVRVW\-M dfl X ( X ) < OO 

and the proof is complete. □ 

Proposition 6.2. The operator Tx is a unitary isomorphism Tx '■ W x — > 
W x with Txipo = ^0 which intertwines the g-action dirx with the Q-action 
dvr A o Ad(j). 
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Proof. By Proposition l3.6l the operator 7a intertwines the Bessel operator B\ 
with the coordinate multiplication — x and vice versa. Therefore we obtain 



U x o dnx(X) = dn x (Ad(j)X) oU x (6.1) 

for X E n © n. Since n and n together generate g as a Lie algebra the 
intertwining formula (|6.ip holds for all X £ g. Further, for x G O x Q Xx we 
find 



TaVo(x) = 2~ rX / J x (x,y)e- t ^ d» x (y) 
= j l x (-2x,y)e- 2t ^ d//(y) 



-2x) = e - tv(x > = iPq(x). 

Since VF A = d7r A (^(g))^ , it follows that T X W X C d7r A (^(g))7IVo = 
Now, since invariant Hermitian forms on the irreducible infinitesimally uni- 
tary (g, t)-module W x are unique up to a scalar, we find that 7a is a unitary 
isomorphism. □ 

Theorem 6.3. U x = Tx- 

Proof. By the previous proposition Ux ° T^ 1 extends to a unitary isomor- 
phism L 2 (Ox, d^x) — > L 2 (Ox, dfix) which commutes with the g-action d7TA- 
Therefore, by Schur's Lemma, Ux is a scalar multiple of T\. Since U\ipo = 
tpo = Txipo this gives the claim. □ 

Remark 6.4. The group G is generated by the conformal inversion j and 
the maximal parabolic subgroup P := L° X N, where 

L° =GnStr(V). 

Write P for the preimage of P under the covering map G — > G. The restric- 
tion of 7Ta to P factors to P and is in the Schrodinger model quite simple. 
In fact 

ttaKM*) =e i ^ a U(x), n a eN, 

TTx{g)ip(x) = x(g*)^{g*x), g£L°, 

and by Mackey theory this representation is even irreducible on L 2 (0a, dfix). 
Therefore, together with the action of j in the Schrodinger model (see The- 
orem ESI this describes the complete group action ttx- 
For the case of the rank 1 orbit 0\ Theorem 16.31 was shown in [231 Theorem 
4.3]. Earlier contributions to special cases are [33] for the case g = so (2, k) 
and A the minimal discrete Wallach point and |37j for the case g = sl(2, R) 
and arbitrary A G W. 
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Remark 6.5. Since the functions x i— > 3^\{x,y), y G O x , are eigenfunctions 
of the Bessel operator, the inversion formula for U x gives an expansion of 
any function if) E L 2 (O x , dfj,\) into eigenfunctions of the Bessel operator: 

^(x)=2- rX [ J x (x,y)U x ^(y)dfi x (y), i> G L 2 (O x , d/x A ). 

Compared to the action of j in the Schrodinger model its action in the 
Fock model is extremely simple. Define (—1)* on Tx by (— l)*F{z) = F{—z). 

Proposition 6.6. M x oU x = (-l)*oB A . 

Proof. We have dp A (i(e, 0, -e)) = ckf (2it(0, 1,0)) = 2it(d z + There- 
fore, we obtain 

p A (e' (e '°'- e) )F(z) = e irXt F(e 2it z). 

For t = ? we obtain the action of j which is given by e lw ~(— 1)*. □ 

This now allows us to compute the action of the unitary inversion oper- 
ator on the Laguerre functions introduced in Section 11.81 

Proposition 6.7. 

= (-1) H &- 

Proof. Since B A £^ = const -$ m and $ m (— .s) = (-l)l m l$ m (z) the claim 
follows with Proposition 16.61 □ 

6.2 Whittaker vectors 

The integral kernel of the unitary inversion operator provides us with ex- 
plicit Whittaker vectors for the representations 7r A on L 2 (O x , d/i A ) and hence 
with explicit embeddings into Whittaker models. For this let L 2 (O x , d/i x )°° 
denote the space of smooth vectors of the representation ir x endowed with 
the usual Frechet topology. Since tt x leaves L 2 (O x , d// A )°° invariant the 
representation extends to the space L 2 (O x , dfi x )~°° = (L 2 (O x , d^ A )°°) / of 
distribution vectors by 

(ir x (g)u,tp} := {u^^g' 1 )^) , g G G, 

for u G L 2 (O x , d^ A ) -00 and tp G L 2 (O x , d/x A )°°. Here we use the com- 
plex conjugate U of an operator U on L 2 (O x , d^ A )°° which is defined by 
Vlp := F^for ip G L 2 (O x , dfi x )°°. Further recall that W x = V{O x )e~ tr ^ C 
L 2 (O x , dfi X ) is the underlying (g,t)-module of ir x . It does not carry a rep- 
resentation of the group G, but the representation d7r A of the Lie algebra q 
restricts to W x . By duality d7r A extends to the algebraic dual (W x )*: 

(dirx(X)u,<p) := -(u,dirx(X)<p), X G fj, 
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for u G (W x )* and (p G W\ We have the following inclusions: 

W X C L 2 (O a , d^) 00 C L 2 (O a , d/x A ) C L 2 (O a , d^ A )-°° C (W A )*. 

Note that the unitary inversion operator IA\ leaves each of these spaces 
invariant. 

Definition 6.8. A distribution u G (W x )* is an algebraic n- Whittaker vector 
of weight i] G (resp. an algebraic xi- Whittaker vector of weight r\ G n^) if 

d7TA(^)u = r](X)u, for all X G n (resp. X G n). 

If moreover u G L 2 (0\, dfix)~°° we call ti a smooth Whittaker vector. 

For 2 G Vc let 7y 2 G and rj z £ be defined by 

rj z (u, 0, 0) := i(u\z), u G n c , 

rj 2 (0, 0,v) := —i(y\z), v G nc- 

Now fix A G W. In |41L Theorem 1] it is shown that there can only 
be non-trivial algebraic n- Whittaker vectors (resp. n- Whittaker vectors) of 
weight 7] G (resp. rj G n£) for d7rA if ?7 is contained in the associated 
variety of d7rA in 0^. The intersection of the associated variety of d7rA with 
n£. (resp. n^) is equal to X\ (after identifying and with Vc)- Hence the 
existence of a non-trivial algebraic n- Whittaker vector (resp. n- Whittaker 
vector) of weight r/ 2 (resp. rj z ) implies z G X\. We prove the converse of 
this statement: 

Proposition 6.9. Let z G X\. 

(1) The Dirac delta distribution S\, z at z is contained in (W x )* and defines 
an algebraic n- Whittaker vector of weight rj z . 

(2) The distribution 

&\,z( x ) '■= J\{ z ^ x )^xi x )^ x e O x , 

is contained in (W x )* and defines an algebraic ri-Whittaker vector of 
weight rj z . 

(3) The algebraic Whittaker vectors 6 x ,z an d <p\,z o^e related by 

<f>x, z = 2 rX U x S x , z . 

Proof. (1) Since W x = P(0 A )e- tr( ^and V(O x ) = C\Xx\ the point eval- 
uation of every tp G W x at z G Xx is well-defined. Now d7TA(u, 0, 0) = 
i(u\x) and the Whittaker property follows. 
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(2) By Proposition lBTTl the function 6\ belongs to (W X Y . Since d7TA(0,0, v) = 
i(v\B\) the Whittaker property follows from Proposition 13,61 

(3) By Theorem 16.31 we have for ip £ W x 

(4>x, z ,f)= / J\(z,x)(p(x)dfj,\(x) = 2 rX U\(p(z) 
= 2 rX {5 X)Z ,Uxv) =r x (U x 5 X)Z ,ip) 
since U\=U\. □ 

An interesting question is to determine the set of smooth Whittaker 
vectors for tt x . We find that the algebraic n-Whittaker vector 5 Xz (resp. 
the algebraic n- Whittaker vector (j) X . z ) is smooth if z £ O x . For this we 
first prove estimates for 5 XjZ and <j)\ iZ using the estimate for J x {z,w) by 
Nakahama |45j stated in Proposition 13.21 

Lemma 6.10. For each A £ W and x £ O x there exist elements 1,7 6 ^(b) 
such that for all ip £ W x we have 

\{8\,x,<p)\ < HdTAWvlUa^.d^A)' ( 6 - 2 ) 
\(<t>X,x,<f)\ < \\dTrx(YM L 2 {0xAtlx) . (6.3) 

Proof. We first prove (|6,3p . From Proposition 13.21 it follows that there exists 
constants C\ > and k £ No such that 

\Jx(x,y)\ <C x {l + \y\ 2 f yyeO~ x . 
Now there exists an N £ No with 

C 2 := ||(1 + | •| 2 ) fc " Ar |lL 2( A ,d MA )<oo. 
Hence we obtain for <p £ W x , using Holder's inequality: 

\(4>x,x,<p)\<Ci [ (l + \y\ 2 ) k \p(y)\dfi x (y) 

= d f (i + \y\ 2 ) k ~ N • |(i + \y\ 2 ) N <p(y)\ tyxiv) 

<CiC 2 ||(i + |-|T^(o.,d^)- 

Since div x (U(Q)) contains all polynomials there exists X £ U{q) such that 

dir x (X)<p( y ) = dC 2 (l + \y\ 2 ) N y{y) 
which shows (j6.3[) . Now by Proposition 16.91 (3) we have 

6 X>X = 2^ X U X ^ X 
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and we obtain 



\{6 x , x ,ip)\ = 2- rX \{U x ^ x ^)\ = 2- rX \((j )x>x ,U\v)\ 
<2- rX \\d7i X {X)U^\\ L 2 {0 ^ ) 
= 2- rX \\dn x (X)TT X (jM L2(0x ^ x) 
= 2~ rX \\TrG)dn x (Ad(r 1 )X)p\\ LHOx ^ ) 
= 2- rX \\d7r x (Ad(r 1 )XM LHOxidlMx) . 

Hence follows with Y = 2~ rX KdQ-^X. □ 

Theorem 6.11. Let A G W. Then for every x G O x the algebraic Whittaker 
vectors 5 XjX and 4> XiX are smooth, i.e. 5 XiX ,cf) X:X G L 2 (O x , d/iA) _ °°- 

Proof. Recall that for X G U(q) the map 

L 2 (O x , d/i A ) [0,oo), ip^- || dTT X (X)ip\\ L2{0xydflx) 

is a continuous seminorm on L 2 {O x , d/x^) 00 • Since W x C L 2 (O x , d//^) 00 is 
dense the claim now follows from (16.21) and (16.31) □ 



Remark 6.12. For x G O x it is easy to see that Ja^j 0a,x G L 2 (O x , dfi x )~°°. 
In fact, by the Sobolev embedding theorem it follows that there is a continu- 
ous linear embedding L 2 (O x , d/^) 00 ^ C°°(Oa) and hence it is immediate 
that 5 X , X G L 2 (O x , d^ A )-°°. Further ^ = 2 rX U x 6 x , x G L 2 (0 A , d^)^ 00 
as IA X is an automorphism of L 2 (O x , d/iA) -00 . However, it is a priori not 
clear that every function ip G L 2 (O x , d/iA)°° extends to 0a and that 5 XiX is 
defined on L 2 (O x , d/^) 00 for x G dO x . In [371 Theorems 5.15, 5.18 & 5.19] 
the statements of Theorem 16.111 are shown for the special case V = M, i.e. 
0^s[(2,R). 

Remark 6.13. Note that since J^ X (0,y) = 1 for all y G V we have <^>a,o = 
IdiiA) the constant function on O x with value 1, and hence 

U x 5 Xfl = 2 rX ld^ x . 

This formula resembles the fact that the Euclidean Fourier transform maps 
the Dirac delta distribution at the origin to a constant function. 
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7 Application to branching problems 



7.1 Branching for so(2, n) \ so(2, m) © so{n — m) 

For g = so (2, n) we consider the representation d7r^ 2,n ^ belonging to the 
minimal non-zero discrete Wallach point A = | = ^y^- We consider the 

restriction of d-7r^ 2 ' n ^ to the symmetric subalgebra t) = so(2,m)©so(n — m) 
of g. Note that the pair (g, ()) is a symmetric pair of holomorphic type and 
hence the restriction of the lowest weight representation d-7r^ 2 ' n ^ to f) is 
discretely decomposable with multiplicity one and the representations of f) 
appearing in the decomposition will again be lowest weight representations 
(see [32]). 

Consider the Jordan algebra V = IR 1 '"" 1 which is the vector space ffi x 
M™" 1 endowed with the multiplication 

(xx,x') • (j/i,!/') := (zi2/i + x ■ y' , x x y' + yix) 

for x\,y\ G M, x',2/' € where x'-y' denotes the standard inner product 

on We have dex(V) = so(n — 1), acting on the last n — 1 variables, 

and stt(V) = 50 (1, n — 1) © BL The conformal Lie algebra of V is given by 
g = so(2,n). The subalgebra 5o(2, m), 1 < m < n, can be viewed as the 
conformal Lie algebra of the Jordan subalgebra U = {x G V : x m+ i = . . . = 
x n = 0} = H. 1,m_1 . Then so(2,m) consists of the elements (u,L(a) + D,v) 
with u,a,v G U and D G 50 (m — 1) acting on the coordinates X2, ■ ■ ■ , x m . 
The centralizer of 5o(2, m) in g is so(n— m) acting linearly on the coordinates 
Xm+i) Hence so(n — m) C fi [ = so(n — 1). Finally let fj = so (2, m) © 

50 (n — m) C g be the corresponding subalgebra of g. 

To study the restriction &ir B °^ 2,n ^\t> we use the Fock model ( dp s ^ 2 ' n \ T\) 
of the representation dTi s ^ 2 ' n \ In this case 

X M2,n) = {zeC n :z 2_ z 2 ^ = 0} \ {0}, 

X?™ = {zeC:zl-4 4/0}, 

realized in the ambient space Vq = C n , and hence 

r{x M2,n) ) = c[Zi) Zn]/{Z 2 _ Z 2 Z 2 ); 

V(X* 0{2 ' n) ) = C[Z 1 ,...,Z n }. 
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For arbitrary A £ W the representation dp*£ of gc acts by 



dp\(a, — 2iL(a), a) = 2i QjZj, 



n 

dp\(a,2iL(a),a) = — 2i ajB™' J , 

i=i 

dp A (a, 0, -a) = 2z ^<9 az + ^ tr(a 
dp A (0,L>,0) = -8 Dz , 

where 



5 J +1 for j = 1, 



s 



dzj J y-\ ior2<j<n. 

We first consider the action of the compact part so(n — m) which acts 
naturally on the coordinates z m _|_i, . . . , z n . Note that 

C[Zi, . . . , Z n ] = C[Zi, . . . , Z m ] <g> C[Z m+ i, . . . , Z n ]. 

Every polynomial in Z m+ \ , . . . ,Z n can be written as a sum of spherical 

harmonics multiplied with powers of (Z^+i + ' ' ' + %n)- m V{Xi°^ 2 ' n ^) we 
have Z^ +1 H \- Z% = -Z% + Z$ -\ h Z^ and hence 

p( ^p(2,„) ) ^ c[Zij _ j ZJ/(Z 2 _ Z | Z 2 } 



oo 



0c[z 1 ,...,z m ]®#H- 

fc=0 

which gives the decomposition into irreducible so (re — m)-representations. 

Next we have to examine the action of so(2, m) on each of the summands. 
For this we use the notation V c = C n = C m C n ~ m = U c U£. 

Lemma 7.1. For k £ No and X £ so(2, m) roe /i<roe 

i so(2,n) / v -\ I i 5o(2,m)/ v -, 

d PA WIC[2i,...,Z m ]®^(C»-™) - d />A+fc W<8>ui. 

Proof. On C[Zi, . . . , Z m ] <g> TL k (C n ~ m ) we check four exhaustive cases sepa- 
rately: 

(1) X = (0,D,0), D G 5o(m - 1). The action is given by 
since D only acts on the coordinates Z2, . . . , z m . 
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(2) X = (a, —2iL(a),a), a € U. Since a£f/C C m it is immediate that 

m 

dp?™ (X) = 2* £ a jZj = dp*°g m) (X) ® id • 

(3) X = (a, 0, —a), a £ U. Assume first that a\ = 0. Then L(a) annihi- 
lates C n ~ m and hence 

For a = e we consider X = (e, 0, — e) which acts by 

dp *°(2,n)pQ = 2i ( E n + X ) = 2i(E m + X + k) 

since E n ~ m = £™ =m+ i z j£j acts on ^ fe (C n " m ) by the scalar fc. 

(4) X = (a,2iL(a),a), a £U. It suffices to check the claimed formula for 
a = ej, j = 1, . . . , m. We then have 

dp s ° {2 ' n) (X) = -2i (e jZj n n - 2{E n + A)^ 

= -2i (ejZjU™ - 2{E m + A + 

= d P *°$; m \x)®id 

since 

^ n [f(zi, z m )g(z rn+1 ,. . . , z n )) 
= CT/Oi, ...,z m )- g(z m+ i, ...,z n ) 

+ f( Zl ,...,z m ) ■ A n - m g(z m+1 ,...,z n ), 

where A n ~ m denotes the Laplacian on C n ~ m and E n ~ m the Euler 
operator on C n ~ m which acts on T-l k (C n ~ m ) by the scalar k. □ 

This shows the following theorem: 

Theorem 7.2. For A = the minimal non-zero discrete Wallach point 
for so(2,n) we have 

oo 

d7r ,o(2,n) = dir s x f k m) ®H k (R n - m )- 

fc=0 
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For m < n we have 



A + k = 



n + 2k-2 



m — 2 



k G N 



2 



> 



2 



and hence the representations dir^}^ belong to Wallach points in the 
continuous part of the Wallach set. For 2k > 2m — n the representation 
d7r^,'"^ are holomorphic discrete series. In particular, if 2m > n then 
there occur representations in the branching law which are not holomorphic 
discrete series. 
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